It’s been about six months since I joined a new company, and I’d like to share what I’ve been working on. Right from the start, my main focus has been using Spring Batch to move critical processes away from manual spreadsheets into an automated, reliable, and scalable pipeline.

In a nutshell, we receive a high volume of data every day—around 180,000 records—that was previously exported to Excel, reviewed manually, and sent back for corrections. This workflow introduced errors, took hours of manual effort, and didn’t allow for scaling. My mission was to build, in Java, a pipeline that:

Reads data directly from the database (no more intermediate files).

Validates and groups each record efficiently, leveraging Spring Batch’s chunk-based processing to avoid out-of-memory issues.

Writes first to a “staging” table, tagging each record as valid or error.

Loads validated records into the final “fact” table at the end of the job.

For those who haven’t used it, Spring Batch is a framework that splits large volumes of information into smaller “chunks” for batch processing—ensuring that failures in a few records don’t crash the entire job. It also logs every step (when a job started, how many items were processed, how many were skipped due to errors), which is invaluable for auditing and traceability.

Meanwhile, I also built a lightweight Angular portal so that business users can quickly see the records that failed validation—without having to open a massive .xls file. They get a clear view of why each record failed, make corrections on the spot, and trigger a reprocessing that, within seconds, feeds the corrected data back into the pipeline. This has cut about 95% of the time previously spent opening spreadsheets, reviewing rows one by one, and emailing files back and forth.

Another key improvement was setting up a Continuous Integration/Continuous Deployment (CI/CD) pipeline. We replaced the “hand-held” FTP deploys with a workflow where every commit triggers a build, runs unit tests, and—if everything passes—executes an automated script on the server. The deployment process includes backing up the previous .jar and running a health check so that, if something goes wrong, we can automatically roll back to the last known good artifact. As a result, what used to take hours now takes no more than ten minutes—and with zero risk of “breaking” production.

In six months, I’ve seen firsthand how a digital transformation built on Spring Batch, Java, and Angular brings speed and reliability to the day-to-day. It’s not just about writing code; it’s about improving workflows for both business and IT teams, providing real-time visibility into data, and freeing up people to focus on value-adding tasks instead of spreadsheet maintenance.

If you’re still relying on manual Excel processes and need to scale to much larger data volumes, I highly recommend checking out Spring Batch.