Kafka - The Architecture Game Changer That You Have Been Looking For

The search for ways to optimize online solution speed performance is a never-ending endeavor with Kafka being a solution that will increase both your speed and productivity

What is Kafka?

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Here is what it is in common terms: Right now the one thing that developers spend a lot of time on is integrating data solutions into their project. They could be creating a solution that takes completed orders and move them to a fulfillment database. They could be take the email_addresses and adding them to a drip campaign. They could be taking the names and address for their customers and adding them to their contact database.

Kafka changes that dynamic by offering a solution that watches the entry of that information into any file format and then processes it for entry into those respective databases I talked about earlier. From my perspective, there are two major upsides, one is the speed of development because connection are plentiful and new ones are being created all the time. The second is that you can skip a lot of the programming that would be required to move this data and just let Kafka handle it for you.

The downside is that there is a learning curve for Kafka but once you understand it, the sky is literally the limit. You can create solutions that will move data around in seconds without the need for programming.

Make Sure You Are Threading The Needle For MySQL Queries

In the history of creating applications, aside from the aesthetics and functionality there is one common thread I have seen affect the success of a website or app. That common thread is speed. The one inevitable issue that will arise for any application is that it will increase in size. This is especially the case if the business is growing and catering to additional customer segments.

One of the items at the center of speed are your queries. How fast are you able to process those queries and provide feedback back to the customer.

Don't ever fix the problem, make sure you fix the customer.

If you goal is really to take care of the customer, you will do what needs to be done to optimize your queries and how quickly they return data.

From a programming perspective, its easy to write a query to get the data. Its another thing to dive into the performance of those queries to see how they are performing. There are three layers of MySQL that you can optimize: the utility layer which controls authentication and connection features, the SQL layer which packages queries for processing, and the storage layer. The more you dive into these aspects of the MySQL database, the faster you are able to provide results to customers.

You should never accept the default results of queries and always look to optimize all three layers. This will give you an advantage that other developers and DBA's are not looking at to improve the Customer Experience.

What I Look For in ERP data

All processes within an organization are going to be around or have some involvement with data. The data could be in the format of information you have and information you do not have. The persistent outlook should have a data science component where you calcuate if you actually need some of the data you have or if you need more. This is rabbit hole territory.

In reviewing the data that is entered in the ERP system, here are some things that I look for in the data.

• What are the consistent patterns in the data? This gives me the opportunity to either optimize the values entered, streamline the interface for users to make their lives either, or automate the process entirely. The best data is the one you dont have to enter.
• What is the velocity of the data? How often if the data entered. This helps identify patterns based on the users and not the data. People are a critical component in the data analysis and data science process. If you understand how they are doing their work, you understand the context of the data being entered.
• Is there data overlap? This provides and opporunity for data re-use and ultimately provide a uniform approach to ERP projects. Wouldn't it be helpful if you could gauge projects based on normalized criteria. You can stop the estimation work and focus on closing business.
• How does the data roll-up? Yes, you can structure data in a way where the reporting roll-up outlines a clear picture of past and current trades.

Dire Warning

Please do not implement any data systems that do not allow you direct access to your data. This sounds like a mouthful but you will curse the solution and try to work to get it to co-mingle with other integrations that you have. The more you push, the more it pushes back because you are trying to get access to your system and the provider is locking the data down behind third-party integrations. This is a level of suck that you do not want in your org. I warned you.