Magento is one of the most well-known and established software e-commerce platforms for creating online stores, the main advantage from a technical point of view is the database design that offers more flexibility to eCommerce managers.

At the same time, this is the main reason for some other problems that occur such as the high consumption of many resources from web hosting servers, if you use a simple web hosting environment you will notice this kind of problem immediately.

The software comes with the community edition which is free, but for the enterprise version of the Magento edition that includes all the advanced features, you have to buy a license from adobe.

Your eshop to run smoothly requires a lot of resources and without a problem so getting consulting from magento web developer maybe the first step. Before we address the issues let's take a close look at the schema database design that is called  EAV data model. This type of database design offers amazing flexibility but at the same time, the trade-off on this that requires more resources.

Initially, the EAV database schema was used for categories data that often need records in medicine and healthcare systems data are sparce and with large number of attributes and a small subset of this attributes are applied to each entity.

  1. Model view controller architecture
  2. Entity attribute value data model
  3. Products types
  4. Price Generation Process
  5. Quote system
  6. Shopping Cart Price Rules
  7. Shipping Integration
  8. Customer Groups

What is EAV data modelling technique

Entity attribute value data model is used to alter the schema of the database without actually altering the tables of the RDBMS system. This is because the columns of the schema are stored to the relational database table, this is used extensively in the area of product's attributes per category.

The EAV is a data model that is used in Magento to provide flexibility in the description of characteristics of a product or a collection of products. So having this flexibility you do not have the need to change the structure of your database design in any new attribute that is added by the merchant.

  • Entity represents data items and the records are stored in table eav_entity_attribute and there are eight types of these records
    • customer
    • customer address
    • catalog category
    • catalog product
    • catalog product
    • invoice
    • creditmemo
    • shipment
  • Attribute are data items that describe entity records like SKU, description
  • Value is data that are taking and save values from a set of values for this attribute

A simple ER diagram on this model and the attributes are stored in database design is the following, so you can understand basic definitions.

Despite the advantage of having this type of flexibility one of the downsides is the system may have a slow performance on high traffic will have to be supported by cache mechanism

Entity Relationship Diagram of Entity Attribute Model tables in magento database

In this case, you can set up specific rules like discounts that can apply in specific categories of the catalog or to products with specific attributes

The role of the attributes set is to associate all the characteristics to product type like the Shoes. The attributes groups are useful to group the attributes so we can find it easier in the administration panel

In the admin panel to take a look at the attribute default set will have to go

Admin -> stores -> Attribute Set -> Default

product attributes in magento admin panel

Model View Controller Architecture (MVC)

Magento follows the ORM approach, where there are two types of models (M) from MVC pattern. The first one is the simplest one is the pattern of Active Record, which means that a model is a class for an object that represents data record from only one table. So if want to have a list of records you have a query a collection of models so when you create a new model

Usually, there are three files to manage data from the database

  • Model extends \Magento\Framework\Model\AbstractModel
    this file contains the database logic does not execute SQL queries
  • Resource Model extends \Magento\Framework\Model\ResourceModel\Db\AbstractDb
    A resource model inherits and contains specialized functions to fetch data
    namespace Vontikakis\DataFirst\Model\ResourceModel;

class DataBridge extends \Magento\Framework\Model\ResourceModel\Db\AbstractDb
{
	
	public function __construct(
		\Magento\Framework\Model\ResourceModel\Db\Context $context
	)
	{
		parent::__construct($context);
	}
	
	protected function _construct()
	{
		$this->_init('vontikakis_datafirst_databridge, 'databridge_id');
	}
	
}
  • Collection Model extends \Magento\Framework\Model\ResourceModel\Db\Collection\AbstractCollection

CRUD

Magento\Framework\EntityManager\Operation\Read

Magento\Framework\EntityManager\Operation\Create

Magento\Framework\EntityManager\Operation\Update

Magento\Framework\EntityManager\Operation\Delete

The loading etc process for EAV models are handled by

CRUD in Attributes

Magento\Framework\EntityManager\Operation\AttributePool



Magento\Eav\Model\ResourceModel\CreateHandle

Magento\Eav\Model\ResourceModel\UpdateHandler

Magento\Eav\Model\ResourceModel\ReadHandler

Loading and saving objects or lists of objects (records) from the database is the same for flat tables on EAV tables. This using a Repository, a Class file under a Resource Model e.g. CustomerRepository with at least four common methods to handle. These are loading and saving data as objects and are getById, getList, save, delete. The repository class also makes easy to search and filter data from tables by combining the getList method and injecting a search criteria object.

 

eav_attribute table of Magento Database

Let's take look at eav data again, one of the most important columns at eav_attribute table is the backend_type. For example has a value decimal for attribute price, that means it will store data to table catalog_product_entity_decimal except in case of the static value.

Entity Attribute Value Data Model (EAV)

At EAV there are 3 types of models, backend models that perform operation like create, update, delete, read, existing operations and validation of attributes. The source model that provide the list for dropdown and multi-select attributes. Finally, the frontend model that map the attributes to the template.

If we want to add new attribute to a product like a multi-select field usually we have to make a Database Migration that is used to change the structure of a database but as we previously mention EAM model exist in a fixed number of MySQL tables in the sense of a virtual DB, so we only will add some data that describe columns and datatype using a migration script.

class InstallData implements InstallDataInterface
{
   protected $eavSetupFactory;

   public function __construct(EavSetupFactory $eavSetupFactory)
   {
       $this->eavSetupFactory = $eavSetupFactory;
   }
   public function install(ModuleDataSetupInterface $setup, ModuleContextInterface $context)
   {
        $eavSetup = $this->eavSetupFactory->create(['setup' => $setup]);
        $eavSetup->addAttribute(
			\Magento\Catalog\Model\Product::ENTITY,
			'recommended_price',
			[
                            'type'          => 'decimal',
                            'label'         =>  'Recommended Price',
                             ......
			]
		);
   }

Migration is like a versioning tool to track changes in a database but in this case, it is a version of an EAV model. We will execute this script using a command from the Magento CLI tool so we perform this installation task.

php bin/magento setup:upgrade

After we add this attribute programmatically the first thing that will happen to the system is to add a new row record at table eav_entity_attribute. This new record will be also added to all sets of attributes. The concept of where the data are stored is related to the backend_type column from the eav_attribute table. Now if we want to make customization in some attributes we will have to override the afterSave method of entity attributes resource model using a plugin.

Also to remind attributes are like column data from product category. Loading and saving process is like flat tables that was described from the following interface so a flat table class extend this

Magento\Framework\Model\ResourceModel/Db/AbstractDB

Adding new attributes to flat models (columns to DB) is more difficult and complex than eav tables approach. There are 3 ways to do this with Getters and Setters, create a new table with an entity with 1-to-1 relation to original class or substitute the original class of entity. This problem is solved by default so you do not have to change schema DB in EAV and corresponding class for loading and saving EAV models is.

Magento/module-eav/Model/Entity/AbstractEntity

In the flat table, the model's load process is complete when the first row once the initial row is complete but in the case of EAV a series of operations have to be executed to load data from attribute related tables with method _loadModelAttributes using a series of union queries and for saving process _collectSaveData.

When to use EAV architecture

There is functionality in admin to change the storage between EAV and flat tables, to remind in the case of EAV tables the data are stored vertically so it's best to use EAV in a new entity when

  • The scope should be included in Entity
  • Other modules or Managers could add new attributes to an entity from the admin panel or change the data types to existing one
  • The number of attributes for a specific entity needs to be more than 1017 because in a flat table in MySQL this is the limit of number columns in a table.
  • If the indexes of INDEX type need to be more than 64

While this architecture provides amazing data flexibility but MySQL wasn't designed to work this way so the overheard in server resources is higher, so at least you should use a vps for hositng.

Magento Products Types

As we mention above attributes describe characteristics of a product there are some default attributes one of the most important is the Price. The price and its generation process are subject to many rules that can be adjusted by managers in the admin panel. First, before dive in pricing let's mention the 6 types of product

  1. Simple Products: the simplest form of a product a single SKU it is physical that means has weight.
  2. Virtual Products: usually are not physical products like subscriptions, membership, services, etc.
  3. Configurable Products: this type is a collection of simple products with different options such as size and color in our SaaS these options we call them variants of a product.
  4. Grouped Products: when you want to buy products in groups within a lower price it all the products on the same page but the customer has the option of choosing the quantity of each product in the list
  5. Product Bundles: can be purchased only all together with the price, SKU, quantity can be fixed or dynamic.
  6. Download Products: this a virtual product ins some in digital format like video, software when someone makes a purchase he can download it from a link

If you want to obtain a specific type of product this is stored to catalog_product_entity.type_id field so when we query the catalog in ORM functions we can filter the collections searching for only virtual products using

 

$productCollection->addFieldToFilter('type_id', 'virtual');

 

Final Price is the one that is used when the products are added to the cart and also depends on the product type. After we see the source code for the final price let's give some definitions of the price

Magento\Catalog\Model\Product\Type\Price

  • Configurable Price: is the default price for all options of a simple product but you can set up different price to children products like XL size item may higher price than a medium size item
  • Downloadable Price
  • Grouped Price: It is a set of simple products where if the purchased all together may exist a lower total price compare to the case that they purchased individually
  • Bundle Price: This is similar to grouped products but with the difference that the seller can choose from options (size, color) to add to the cart.
  • Catalog Rule Price Modifier: In this can set up specific rules like discounts, start and end date that applies the rules, application in specific categories of the catalog or to products with specific attributes
    • Base Price
      • min Special Price: In this case a price with discount in a fixed time duration
      • min Tier Price: in this case, a discount can be applied to the product if the user buys above the lower quantity of this product

Price Generation Process and Concepts

There are various methods to change and modify the price process

  • New Virtual Price Type. A virtual type is a way to change the dependencies and arguments that are injected into an instance of an existing class, a custom constructor. It is a constructor dependency injection that is defined at the di.xml file. Let's see an example where we want to add a new field called Recommended Price in admin, first, we will add new price as an attribute as described in the previous section "EAV Attributes in general structure" using an installData class. Assuming that we did that we will add a new price by extending BasePriceProviderInterface and AbstractPrice class so we have an existing class with custom arguments. 
    use Magento\Framework\Pricing\Price\AbstractPrice;
use Magento\Framework\Pricing\Price\BasePriceProviderInterface;

class RecommendedPrice extends AbstractPrice
{
    const PRICE_CODE = 'recommended_price';
    
    public function getValue()
    {  ...
    }
}

    <?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:ObjectManager/etc/config.xsd">
    <virtualType name="Magento\Catalog\Pricing\Price\Pool">
        <arguments>
            <argument name="prices" xsi:type="array">
                <item name="recommended_price" xsi:type="string">Vontikakis\DataFirst\Pricing\Price\RecommendedPrice</item>
            </argument>
        </arguments>
    </virtualType>
</config>
  • Plugin for overriding methods. Plugins in Magento are interceptors, which means that intercept and change the behavior of public methods, this on a class without inheritance. A good use case is when we want to transform data. This technique that changes functionality to methods in another PHP framework like the Yii framework is named Behavior and the general term in software engineering is mixin

    Let's take an example where the store owner wants to add $3 to all units. The scenario that the owner wants to offer "free returns and replace" so the products may cost 3 units higher. Such modifications are done in method getPrice()

    namespace Vontikakis\DataFirst\Plugin;
use Magento\Catalog\Model\Product;

class Product
{
    public function afterGetPrice($subject,  $result)
    {
        return ($result+3);
    }
}

    If you notice in the above piece of code the modification has down by adding the word after before the name of the original method that was modified. So we created an after type plugin that modify the return value of method. Also there two other types of plugins the before type that modifies inputs and type around that is used on complex validations cases, and last but not least do not forget to add the plugin to di.xml file and do not forget that can be used only when modifying public methods. Plugins are useful to transform data and should be avoided when there is an event observer there that is to transmit data as we will see in the following sections.

  • Override Specific Type of Product
  • Override the price class

all products belong to the least one category one important thing is that categories are EAV data are stored to table that named catalog_category_entity. For creating new categories programmatically you can you use the object manager to create an object from the class

Magento\Catalog\Model\CategoryFactory

and then implementing

Magento\Catalog\Api\CategoryRepositoryInterface

The table where categories are saved in the database has 2 important fields the path and level

path: is the id of the path starting from number 1 where is the root category and last number entity id

level: represents the depth of the category in the tree

Another important field is the catalog_category_entity is the field

parent_id: Root categories automatically take the value 1 for root categories and id of the level up category.

Another aspect is the catalog price rules, that are applied to products before they added to cart, they are stored to table catalogrule, for apply discounts in customer groups, categories, etc. When the rules take effect the table catalogrule_product_price is populated with necessary data and also there is some cost on the performance of the store the rules are indexed by Catalog Rule Product Index so the debugging we alway to look for problems that inflow or outflow from the indexed table. For adding new rules from Admin Panel -> Marketing -> Catalog Price Rule

Magento Quote System

What is a quote? Quote the process that the system starts to track the activity of a user from the time that starts to add items to the cart until he ends his checkout process by abandoned the cart or place his order. So in quote includes information such as customer contact data, shipping address, the products, the payment method, the shipping method and cost, total price, subtotal and discounts.

So All the tables in the DB that start with the prefix quote_ will store the data

  • quote
  • quote_address
  • quote_address_item
  • quote_id_mask
  • quote_item
  • quote_item_option
  • quote_item_payment
  • quote_shipping_rate

In the bottom line, the quote is the storage of the shopping cart session in the DB. As mention, the quote includes all the relevant information about the order. One key part of this process of Totals. This is defined from the following class

Magento\Quote\Model\Quote\TotalsCollector

This class has some methods that calculate the total some in a quote like the total price of items, the total cost of shipping and the total price of an order per address. We can modify the process and create a custom total model. First we a custom class that inherits from

\Magento\Quote\Model\Quote\Address\Total\AbstractTotal

and then modify and add it the system through /etc/sales.xm

Shopping Cart Price Rules

The shopping cart rules are used to give a discount to the customer at the checkout level like free shipping, a discount on total order under certain condition, this means they are applied after a product added to the shopping cart, on the other hand, catalog rules are applied before products added to the cart. In admin level, you create a rule.

Admin Panel -> Marketing -> Add New Rule

or programmatically if someone wants to have more complex conditions and operations

 \Magento\SalesRule\Model\RuleFactory

The more rules you have the slowest the performance on the system will be like the add to cart events. The add-to-cart functionality is very important to be kept in optimal performance. Regard this someone can add products from 4 sections

  • Product Page from module-checkout
  • Wishlist from module-wishlist
  • ReOrder from module-sales
  • Quotes Merge from module module-quote

In every case, there are triggered a series of events.

The shopping cart adds to cart process depends on the section page that the product is added and also from the type of product.

  • Simple product exist in one line in checkout
  • configurable product inherits the product title and shoe the visible option in one line
  • Bundle Product all the option are visible in the same line
  • Group Products are shown in a multiline data grid

The rendering of product items is defined checkout_cart_item_renderers.xml in various where checkout takes place. Important method to add a quote or remove quotes from the shopping cart is Quote Module methods AddItem, remove Item.

Magento Shipping Integration

Creating new shipping methods can be done by implementing the CarriernInterface and extended

use  Magento\Shipping\Model\Carrier\CarrierInterface;
use  Magento\Shipping\Model\Carrier\AbstractCarrierOnline;

Each carrier like UPS provides a list of shipping rates that are applied under different conditions etc, Fast Delivery. The same time rates may change in different periods some shipping method modules can communicate with carrier API so they can print labels for shipping.

To debug the shipping method and check if the shipping table rates are coming through Shipping Carriers API first we have to check if shipping is enabled then check request and results from collectRates

Magento\Shipping\Model\Carrier\AbstractCarrierInterface::collectRates

You find to Magento marketplace shipping modules for free for companies like UPS or order a custom magento module

The payment methods are 2 types the offline and online methods, the first one includes bank transfers, pay on arrival options and the second are the methods that connect the eCommerce store with an online payment processor like Paypal, Stripe, etc using an API.

Integrate Magento with ERP

Interconnectivity is the key to success to any modern system, if we are able to connect this into party software like ERP systems. There is here a well-documented REST API   and you can retrieve the latest orders and import them to your ERP by query endpoints HTTP protocol. An alternative way to use a broker message system using the amqp-module, If you choose to use REST you will have to send requests to Magento.

In another case instead of request data, Magento will send data after a specific event occurs like place order triggered and send data to third-party software, this can be by creating a plugin for methods placeOrder or submitQuote

Magento\Quote\Model\QuoteManagement

or by creating an event observer like

  • sales_model_service_quote_submit_success
  • checkout_submit_all_after

Creating observers is another key concept in this architecture that someone can use as the above or creating custom implementing the following interface

Magento\Framework\Event\ObserverInterface

The events are defined in etc directory of module in file is named events.xml and also can be isolated to be active in the frontend and backend area.

The general sequence of an order is the following from start to end is this

Processing has 4 steps Order Placed -> Order Invoiced -> Order shipped -> Order Completed

If you want to make modification on the status of orders the corresponding class that hands this functionality is

Magento\Sales\Model\Order

An important part of sales operations is the refunds, here are called Credit Memos, is the process of returning money to clients for returned items. There are two types the first one the offline memo where you marked the order and you handle the return money on your own like bank transfer the money and other type is online, that means if the payment was done through a payment processor like authorize.net there is a button that programmatically the event of refund will occur.

Magento Customer Groups

every client can create his own account login and see the previous sales orders, print invoices. The file that controls the user interface for clients is

view/frontend/layout/customer_account.xml

If we want to add a custom attribute, the process is almost the same as we describe above for any other EAV attributes. Tthe only key difference here is every new attribute in this section must be associated with customer_form_attribute table so this attribute's value could be updated through a form.

Another topic is creating customer groups so you can make segments and offer different prices or discounts. In these groups, for example, if some of your customers are other business you should offer your catalog at wholesale price. Another example is if they have a member's cards to collect reward points from purchases in physical stores. You could give a member's discount with dynamic segmentation using parameters like the location. Even with the level of the total amount of previous order.

Almost every transactional from retailers is subject to VAT tax. Setting up this from admin panel we to setup from admin->stores> tax zones and rates and tax rules and also configure tax classes from store configuration.

Magento as the software is a modular system that is highly customizable. That as result makes difficult to track business logic from 3rd party extensions. In a module addition to Resource Models and Models, EAV or flat, has the concept of service contracts - interfaces. These exist is a set of PHP interface under the subdirectory of Magento/Customer/Api/Data so to keep integrity in the data resources.

Deployment Process after making changes

The deployment process in Magento is necessary to so we can apply in a safe way changes like installing a new module, this can be done using an automated deployment tool like Capistrano or manually in simple steps.

  1. You have to enable maintenance mode - bin/magento maintenance:enable
  2. copy files to server, if you use git as source control system just export the working tree
    git --work-tree=target_path --git-dir=repository_path checkout -f
  3. Enable new module - bin/magento module:enable
  4. Run dependency compilation - bin/magento setup:di:compile
  5. Build JS and CSS files using a task runner like gulp.js
  6. Run the migrations to upgrade the database - bin/magento setup:upgrade
  7. Disable the maintenance mode - bin/magento maintenance:disable

The deployment process heavily relies on cli tools, so do not forget exploring available commands of bin/magento and if you want to add a custom command use the dependency injection to expand the list on Magent\Framework\Console\CommandList

Magento Custom Module Creation

As we know this is a module-based system so a custom module could be installed to the following path app/code. The only 2 files that a module needs into be launched and integrated into the system are registration.php, etc/module.xml.

Then next concept to understand is how the routing system works in custom modules and Magento application but first let's describe how the application is initialized, from the entry point pub/index.php that follows the HTTP applications which is defined the area (frontend, adminhtml) follows the initialization for frontController class which search the list of routes and match corresponding controller action.

How the process in URLs and map them to modules and actions is simple just take a look at the following example url http://magento.localhost/datafirst/lake/index there are three main parts but the code that determines in which is the url belongs adminhtml or frontend is determined by the code Magento\framework\App\AreaList

1st:  datafirst ->  name of module DataFirst

2nd: lake->  name of folder that contains the action in Controller in case of frontend Lake

3rd: index ->  name of action file Index.php extends Magento\Framework\App\Action\Action

In the case the controller belongs to adminhtml area custom module's uri is prepend with the admin keyword and in the end of the is appended a secret key to protect the admin panel

http://magento.localhost/admin/datafirst/lake/index/key/b75c0e672b

Each action returns page, layout, json, raw, redirect, forward

Magento\Framework\Controller\ResultInterface

The first parameter is defined in file /etc/frontend/routes.xml , the second one is name of folder that contains the action and  slashes characters in url (-)  are replaced with underscore in folder directory system, the third and last one is the name of the action that triggered is the execute()  method in this file.   The URLs in adminhtml are almost the same logic the only difference the routes has path is defined  in /etc/adminhtml/routes.xml

Another important note on routing is to create friendly URLs for this there is a table named url_rewrite where the request_path can be mapped to target_path. The module that is responsible for such functionality is the module-url-rewrite and the friendly URL the url_key attribute for product and categories entities. Also, the default 404 page is just the cms page if we want to modify the use NoRouteHandler.

Layouts are XML files that provide instructions on position parts of a webpage  like a header, footer etc and meta info. This kind of files is located inside the theme folder and in the are merged to one big XML file and served a static file by cached directory. The two basic design building block for Magento is the Root Template (root.phtml) and Page Layout that can be found in custom modules under the path of view/layout/page.

As we continue to build custom module roadmap we have to map the controller and action datafirst/lake/index to the layout XML file, for this with having the same name with underscores like datafirst_lake_index.xml , this file is placed in subdirectory DataFirst/view/adminhtml/layout.

After introducing in short the concept of layouts the connected important element of User Interface is the blocks. The block is a PHP class that provides the way for retrieve data where will be printed a phtml files, this is defined inside layouts. A block can contain other blocks and the lifecycle of a block can be found this file

Magento/Framework/View/Element/AbstractBlock

Block created  __construct -> block attached to layout setLayout -> the html is rendered by toHtml -> if cached record exists (cache lifetime is not null or block_html cache is enabled) otherwise  the real html rendering  happens  _tohtml also a blocked is cached in two cases where Cache Life Time is NULL and block_html cache is enabled.

<block class="Vontikakis\DataLake\BlockDisplay" name="block_name" template="path_to_template" />

The above block code exists inside into layouts files and is the connection between the block and a template. Follow is a list of items that you can see inside the layout XML file.

  • <block/> create a block
  • <container/>  a group of block elements
  • <arguments/> & <argument/> pass variables to template files phtml
  • <referenceBlock/> modify an existing block
  • <referenceContainer/> modify an existing container
  • <move/> move a block

Last but not least an integral part of User Interface is the javascript, so when we want to introduce javascript external JS file in template PHTML we can use it as a component with following script tag as

<script type="text/x-magento-init">

to invoke RequireJs module or directly in an inline element data-mage-init

<div id="name" data-mage-init='{"js_module_path_file":{"param":"val"}}'

In admin

1. Custom Modules adminhtml - some key concepts to consider when developing in admin panel

  • access control functionality and access policies are defined in etc/acl.xml file
  • extended classes and implementing Interfaces
  • UI Components like Grids and Forms make is the representation of listing, searching updating data just by combining a model with UI component

Some notes on custom modules especially on when developing in admin. The two important files are configurations

/etc/adminhtml/menu.xml

the definition to add items is <add/> with the following list of attributes

  • id
  • title
  • module
  • shortOrder
  • parent
  • action controller
  • resource - is one the most important parameter in menu item this will map to your acl.xml

Next is the file for the authorization for who accessing the resources in various parts of the site

 /etc/acl.xml

Every menu item has an ID a seen in above is defined in the resource attribute of xml menu item, this ID is checked against the action to a user group for authorization. The access to specific actions from a user group is done using Zend ACL permission component.

After you defined the access permission you can see the access for user role from the following tree list that can be in the admin panel

admin -> System -> Permissions -> User Roles

Also to create new User Role you can use in this class

\Magento\User\Model\UserFactory

Finally, If we want to make our Magento Custom Module easily extensible from others we have to make use of extensions attributes that are a concept similar but other custom EAV attributes. By creating this we allow other to use complex data and a minimum requirement for such as functionality is to add in our module we have to define an interface for each model and add getExtensionAttributes() and/or setExtensionAttributes() methods.

interface CustomerExtensionInterface extends
 \Magento\Framework\Api\ExtensionAttributesInterface { }

in the entity, and also we have to create a file under module subdirectory

Vontikakis/DataLake/etch/extensions_attributes.xml