Now many of you might want to become more efficient and don’t know how – here you go – by automation.
For me, it all usually starts with being annoyed by very repetitive tasks. Those are the tasks that mostly are boring – and I really don’t like boring.
Background of the story
So back in days I had to export certain data-set daily (weekends included!) into the shared folder that was transferred to certain other systems before 8 o’clock in the morning. In e-commerce, there is a great need to have certain lists exported and imported into other systems (for sending marketing campaigns out, for example).
And I didn’t like waking up during weekends. I really didn’t. And there was this other problem – if I was off, somebody else had to do it.
Manual Automation
So the first step from “fully manual” to manual automation was to write a SQL query that uses calculated dates/times instead user changing the SQL query. Yep, this one is easily achievable with some knowledge of SQL:
SELECT *
WHERE “DATE” >= CURRENT_DATE() – 1;
This will give you last 1 day changes in the query.
I also think that part of manual automation are the following steps:
- Putting export files, based on the above statement, to a shared drive
- Sharing those files with people that will manually import it to other systems
- Import those files into the required system
(Full) Automation
This is where you really make a difference – save yourself some (a lot) of time and let the computers do the heavy (repetitive) lifting. You can invest your precious time on the important stuff!
Now here you can go down multiple roads:
- write the automation scripts yourself
- search for automation/integration tool
Write scripts yourself
If you have good programming knowledge (or you think you can Google anything and make a decent version of code) and not really budget to introduce automation/integration tools, you should write your own automation scripts.
If you really want it to be a proper automated solution, you might want to put it into a cloud, which is online 24/7 (or at least configurable to be online when it needs to be online). Besides that, you will need some online storage to put files. And it needs to allow some sort of triggers.
We used Apache Airflow as an orchestration tool and Python scripts (stored within GitHub) to load data between systems. The scripts were connecting to source systems, execute certain SQL statements and move data to (AWS) bucket. The other task in the Apache Airflow was triggered after the previous was finished and that task pushed data from a bucket into the Marketing System.
Automation Tools
Especially in the e-commerce area there are a lot of integration tools that will help you automate your data pipelines.
They will allow you, by drag and drop and clicking, to automate most of your data transfers. They usually defer based on the number of integrations with different systems and tools you are using.
It is essential to write down all your tools and systems you’re using and try to find a tool that supports all or most of them. If they don’t support some, they should allow you to use some language (Python, R, …) to cater for your needs for special systems. And make sure that you can get a proper support.
Testing
I would recommend you spend some time to test and, if possible, you should work on different systems – development, production – to avoid sending wrong marketing campaigns to wrong people.
Conclusion
So how to become more efficient by automation? If you want to become more efficient, you should do it by automation. Automate the repetitive tasks will also make you happier and able to focus on the important challenges. And it can also improve your profit.
And it will make your whole business more scalable and reliable – you will remove the human factor (such as “I forgot to upload file”, “My internet didn’t work”, …)