How to read big json file in php

I need to parse a big JSON file (about 200MB). With json_decode I get a memory error

I have found salsify/jsonstreamingparser library that promise parser JSON documents. But I only have found in-memory examples that produce the same error of json_decode.

Anyone knows how can I parse large JSON files?

asked Feb 13, 2018 at 18:22

11

If you want to iterate over it and do not demand to have the whole structure in memory at once, try https://github.com/halaxa/json-machine. It allows you to simply iterate over a JSON of any size/length just using foreach.

answered Nov 29, 2018 at 14:26

3

Depending on what webserver or php-version you use. It is basicly possible to set a value for max-used-memory and max-used-script-execution-time.

See :

PHP.NET - Limiting Ressources

PHP.NET - Limiting Execution-Time

IMO PHP is not made for this kind of cpu-heavy operations, i would prefer to use it in the more common way - just use a mysql, postgres, ... So parse all your data in your favourite way in your database. Problem solved :)

... hope so. HF

answered Feb 13, 2018 at 18:43

1

I'm working on a project now where I need to ingest and output large amounts of data between systems, without direct access to databases. This has come up on past projects with CSV files, but in this case I am using JSON for various reasons which changes things quite a bit.

CSV files are somewhat easier to work with when dealing with large amounts of data because each record is on its own line. Thus it's easy to create a basic file parser that will do the job by just reading one line at a time. However, with JSON, the file could be formatted in multiple different ways, with a single object possibly spanning multiple lines, or there may be just one single massive line of data containing all the objects.

I could have tried to write my own tool to handle this issue, but luckily somebody else has already solved this for us. In this case I am going to demonstrate the usage of the JSON machine PHP package to process an extremely large JSON file.

Setup

First we need to create an artificial large JSON file to simulate our issue. One could use something like an online JSON generator, but my browser would crash when I set a really high number of objects to create. Hence I used the following basic script to simulate my use-case of a massive array of items that have a depth of 1 (e.g. just name/value pairs).

 md5(rand()),
        "isActive" => md5(rand()),
        "balance" => md5(rand()),
        "picture" => md5(rand()),
        "age" => md5(rand()),
        "eyeColor" => md5(rand()),
        "name" => md5(rand()),
        "gender" => md5(rand()),
        "company" => md5(rand()),
        "email" => md5(rand()),
        "phone" => md5(rand()),
        "address" => md5(rand()),
        "about" => md5(rand()),
        "registered" => md5(rand()),
        "latitude" => md5(rand()),
        "longitude" => md5(rand()),
    ];
}

print json_encode($items, JSON_PRETTY_PRINT);

I made sure to test this with and without the use of JSON_PRETTY_PRINT. This results in the generated file having different formatting, but the end result of this tutorial is exactly the same.

This generated me an 843 MB file of one million items, which I feel is suitably large enough for stress testing.

Running

Now that we have a suitably large file, we need to process it.

First we need to install the JSON Machine package:

composer require halaxa/json-machine

Then we can use it in a script like so:

This doesn't actually do anything that useful. It just prints out each object one-by-one. However it does demonstrate that we can safely loop over all the items in the JSON file one at a time without running out of memory etc. We can take this further and write some code to possibly batch insert them 1,000 at a time into a database or do some sort of operation before outputting to another file.

Last updated: 18th March 2021
First published: 18th March 2021