Popular posts

Bteq Commands

  1)       LOGON:        The first command to perform the database operations is .LOGON. .LOGON generally has four parameters, 1. TDPID 2. User ID 3. Password 4. Acct ID Syntax: Enter your logon or BTEQ command:       .Logon 127.0.0.1/Tduser       Password:       **** Message From BTEQ:       ***Logon Successfully Completed.       ***Teradata Database Release is 12.00.00.10 ***Teradata Database Version is 12.00.00.10 ***Transaction Semantics Are BTET. ***Character       set Name is ‘UTF16’ ***Total Elapsed time was 1 Second. Bteq Logon Elements: Element Requirement Description Tdpid Optional The Teradata Director Program userid Required The User Identifier Password Required The Pwd for your userid Acctid Optional The account identifier for your userid Note:  Omit Acctid ...

Python code to deal with REST API by uploading images

Python code to deal with REST API by uploading images Here we are taking the MulticolorEngine API as an example. Making an API call by uploading images using Python: The image file is sent to the MulticolorEngine API, but only one may be specified at a time and get the dominant colors as response. For each color, a similarity rank, a weight factor and a color class are returned along with the RGB values. We can make the API call either by uploading the image files or by sending the image URLs.  i) Python code using the image files: import requests from requests.auth import HTTPBasicAuth api_url = ' https://multicolorengine. tineye.com/sandbox/rest/ extract_image_colors/ ' user = 'username’ password = 'password' auth = HTTPBasicAuth(user, password) files = {     'images[0]': open('image1.jpg', 'rb') } response =  requests.post (api_url, auth=auth, files=files) json_data = response.json() print(json_data) images[n]:  The image file from which ...

Mapreduce Example

Hadoop Mapreduce : Map Phase : This is a phase whern the Mappers will accept the task and process(division of computation) specific to each node. The result will be in key-value pairs. This is called intermediate output and will be stored in Local disk. Sort and shuffle : Each key value pairs from each mapper are taken and the values are now joined based on Keys and stored in local disk . After sorting and shuffling is done based on Keys of Key-value pair , the values will be sent to Reducers. Reduce :   The output from the sort and shuffle will now be reduced and is stored in HDFS. This will be the final output. Key- Value Pair :  This is the output of the Mapper which will be given for Sorting and merging . Combiner  :  It is called as a mini reducer .It is generally used for searching in data set (Example highest salary in employee table). It will search the highest of each dataset from Map stage . Hive cannot convert nested subqueries into joins Sample Text : ...

Python code to deal with REST API by posting json data

Python code to deal with REST API by posting json data Here we are taking the Cordial API as an example. Making an API call by  posting json data  using Python: To use the Cordial API, you must use their API key and also whitelist the external API of the client that is calling it. Here we have a convenient feature that allows data exports to be directly saved to a GCS bucket. The Code will be as follows:         # Parameters required by Cordial API call         url = ' https://api.cordial.io/v2/ contactactivityexport '         compress = False         all_properties = True         # Start datetime parameter         start_time = execution_date.subtract(hours= 1).strftime('%Y-%m-%dT%H:%M:% S.000Z')         print("start_time api: "+start_time) # This variable is in this format: 2020-04-15T17:00:00.000Z  ...

Hadoop Yarn Architecture

  Yarn ( Yet Another Resource Negotiator) : The  YARN was introduced basically to split up the functionalities of resource management and job scheduling or monitoring into separate processes .The Whole  idea was to  have a global ResourceManager (RM) and for each application an ApplicationMaster (AM). An application can be a single job or a DAG of jobs so that the MapReduce jobs will run unchanged on top of YARN with just by recompile. Resource Manager : There are 2 components in ResourceManager: 1.Scheduler 2. ApplicationsManager.   Application Manager : The ApplicationsManager will accept the submitted jobs and ignore the first container for executing the application specific .Application Master and on failure,it provides the service for restarting the ApplicationMaster container.Per-application ApplicationMaster will ignore the appropriate resource containers from the Scheduler, track their status and monitor their progress. Scheduler :    ...

Teradata Architechture

The Shared Nothing architecture of Teradata means that each vproc is responsible for its own portion of the database and will perform operations on its own environment(a portion of the database). A V-Proc is like an independent component which has its own computer area and storage 


This diagram shows the Architecture of TERADATA and is very useful and easy for learning purposes. Architecture involves three main components which are also called as V-Procs, those are 
  • PE
  • AMP
  • BY-NET

PE (Parsing Engine):

       The PE is a vproc that communicates with the client system on one side and with the AMPs (via the BYNET) on the other side. Each PE executes the database programming that supervises sessions breaks down SQL statements into steps possibly in parallel, and returns the answer rows to the requesting client.

It performs below 4 engine elements.

Parser
·         The Parser look into request catch to determine if the request is already are there (or) not.
·         The Syntaxer checks the syntax of the Incoming request.
·         if errors are present It stops the execution of the request and send an error back to TDP (or) Client.
·         Resolver adds information from data dictionary to convert data base name, table name, view name and macro names to internal identifiers.
·         Security module checks privileges on the data dictionary for incoming request.
Optimizer: 
·         It determines the most effective path for the incoming SQL request. 
Generator:
·         generates transforms the optimized parser tree into plastic steps.
·         It also binds parameterized data and transforms plastic steps into concrete steps.
Dispatcher:
·         It controls the sequence in which the steps are executed.
·         Dispatcher tells whether the request is one AMP, all AMP, and several AMP operations.
GnC Apply will take care of sending the requests to BY-NET .

BY-NET:

Message Passing Layer is nothing but BY-NET.
Its responsibilities include
  • PEs and AMP'S communicate via BY-NET(message passing layer).
  • Sending, Merging and Sorting Result set ( Rows obtained after the query execution).
  • Different communication approaches like 
    • Point-to-Point  :  A vproc can deliver a message to another vproc
    • Multicast : A vproc can driver a message to multiple vprocs by delivering a broadcast message to all nodes 
    • Broadcast : A vproc can broadcast by delivering a message to all the vprocs in the system
  • Dual BY-NETs will be available for Fault tolerant and high Performance

Access Module Processor (AMP):
  • Every Amp listens instructions from PE via By-NET . 
  • Each AMP is allowed to read and write data in its own disk.
The AMP vproc supervises the teradata database interactions with the disk subsystem. Each AMP supervises a share of the disk storage.

 AMP functions include

 Database management tasks
  • Accounting
  • Journaling
  • Locking tables, rows, and databases
  • Output data conversion
During query processing
      • Sorting
      • Joining data rows
      • Aggregation
File management Tasks
  • Disk management tasks in VDisk
Along with above components Please find below three components for better understanding

CLI (Call level interface):

It is a lowest level of interface to the Teradata RDBMS. It handles activities like
·         Helps create session.
·         Allocate request (or) response buffer.
·         Creating Parcels.
·          Fetch Responses.

TDP (TeraData Directory Program):

·          Session initiation, session termination, session balancing (or) managing.
·          Session validation (or) verification and login.
·          providing input to PE'S receiving output from PE (Parsing Engine). 
·         And it handles request ques.

PDE (Parallel Data Base Engine):

PDE is a software layer interface that lies between Operating System and TERADATA Database.
It supports parallelism and gives Teradata database high speed and linear scalability.
PDE is independent of OS like (MPRAS,Linux,Microsoft Windows) It can be able to the below task.
·         Run in parallel environment.
·         Execute VPROCs.
·         Consistently Manage Memory, I/O Messaging System Interface Across Multiple OS.
·         It performs dynamic Distribution of Database task.
·         It performs co-ordination of task execution within and between the nodes.