Showing posts with label streaming. Show all posts
Showing posts with label streaming. Show all posts

PHP memory consumption with Arrays and Objects (update: HHVM, phpng June-2014)

with No comments:
Lessons learned:
  • PHPng is significantly faster and uses significantly less memory
  • objects need more memory than arrays (+ 5-250 percent)
  • if array values are numeric, don't save them as strings!
  • saving 1M integers takes 33M of memory with PHPng (increase by factor 8)
  • saving 1M integers as strings takes 79M of memory with PHPng (increase by factor 20)
  • using SplFixedArray can reduce memory usage by 20-100 percent
  • avoid big Arrays and Objects in PHP whenever possible
    (don't use file('big_file') or explode("\n", file_get_contents('big_file')), etc.)
  • use streams whenever possible (fopen, fsockopen, etc.)
  • use generators when available with PHP 5.5 (RFC)
  • comparing 32bit to 64bit systems, memory consumption increases by 100-230 percent

If you need to save memory temporarily in your script, you can stringify your array (increasing cpu usage and runtime): 

function test(){

$a = array();
for ($i=0; $i<1000000; $i++) $a[] = $i;
$a = implode(',', $a); // or $a = json_encode($a);
echo number_format(memory_get_usage(true)/1048576, 2)."\n";

}
test();
// 7.5 M / 0.64s (php), 10.4 M / 0.55s (hhvm), 7.25 M / 0.20s (phpng)

Here is a small script to show how much memory PHP needs to handle big Arrays:
the total runtime of the script is 15.5s (php), 8.4s (hhvm), 4.6s (phpng)
(tests made with PHP 5.5.9, HHVM 3.2.0-2014-06-06, PHP 5.7.0-dev-2014-06-05, 2.8GHz single-core openvz) 

ini_set('memory_limit', '1024M');

function test(){

$a = '';
for ($i=0; $i<1000000; $i++) $a .= '1';
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 1.50 (php), 3.5 (hhvm), 1.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.25 (php), 30.4 (hhvm), 32.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146.25 (php), 80.92 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = true;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54.25 (php), 30.56 (hhvm), 16.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.50 (php), 30.56 (hhvm), 33.25 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 80.97 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = $i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54.25 (php), 30.58 (hhvm), 16.75 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 61.10 (hhvm), 78.50 (phpng)

$a = new stdclass();
for ($i=0; $i<1000000; $i++) $a->$i = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 237.75 (php), 111.50 (hhvm), 128.25 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 145.5 (php), 61.11 (hhvm), 61.75 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[(string)$i] = (string)$i;
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 61.12 (hhvm), 78.50 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.5 (php), 30.62 (hhvm), 33.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 81.02 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = "";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54 (php), 30.63 (hhvm), 17.00 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138.5 (php), 30.64 (hhvm), 33.75 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 146 (php), 81.04 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = "hello";
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 54 (php), 30.65 (hhvm), 17.25 (phpng)


$a = array();
for ($i=0; $i<1000000; $i++) $a[$i] = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 222.5 (php), 30.65 (hhvm), 34.00 (phpng)

$a = new stdclass;
for ($i=0; $i<1000000; $i++) $a->$i = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 230 (php), 81.05 (hhvm), 82.50 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<1000000; $i++) $a[$i] = array();
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 138 (php), 30.66 (hhvm), 17.50 (phpng)


$a = array();
for ($i=0; $i<500000; $i++) $a[$i] = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 214.75 (php), 18.67 (hhvm), 18.25 (phpng)

$a = new stdclass;
for ($i=0; $i<500000; $i++) $a->$i = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 218.25 (php), 41.81 (hhvm), 41.75 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<500000; $i++) $a[$i] = array("");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 176.00 (php), 30.68 (hhvm), 17.50 (phpng)


$a = array();
for ($i=0; $i<500000; $i++) $a[$i] = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 214.50 (php), 18.68 (hhvm), 18.25 (phpng)

$a = new stdclass;
for ($i=0; $i<500000; $i++) $a->$i = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 218.25 (php), 41.82 (hhvm), 41.75 (phpng)

$a = new SplFixedArray(1000000);
for ($i=0; $i<500000; $i++) $a[$i] = array("hello");
echo number_format(memory_get_usage(true)/1048576, 2)."\n";
// 176 (php), 30.69 (hhvm), 17.50 (phpng)

}
test();

More details and technical explanations about PHP's memory usage can be found on nikic's blog.
More details about PHPng can be found here.

Shortly: a string or an integer in PHP are not mapped to a char-array or an integer in C. A more complex structure is used to get from dynamic typing in PHP to static typing in C: 


struct _zval_struct {
  zvalue_value value;     // value object
  zend_uint refcount__gc; // number of references to variable
  zend_uchar type;        // type of variable
  zend_uchar is_ref__gc;  // reference? (&$var)
};
typedef union _zvalue_value {
  long lval;              // integer and boolean
  double dval;            // float (double)
  struct {
    char *val;            // string (with zero bytes)
    int len;              // length of the string
  } str;
  HashTable *ht;          // array (hash table)
  zend_object_value obj;  // object
} zvalue_value;

Things you should not do in PHP (update: references)

with No comments:
Here is a list of things you should not do in PHP. Most of the stuff is pretty obvious, but over the years I've seen a lot of them. In most cases, these problems remain hidden until data grows above 10000 entries. So on a development system, things are always fast and there are no problems with memory limits :-)

Suppose we have a table with 100k entries: 

$db->query('create table stats (c1 int(11) primary key, c2 varchar(255))');
$db->query('begin');
for ($i=0; $i<100000; $i++) {
  $db->query('insert into stats values ('.$i.','.($i*2).')');
}
$db->query('commit');
Populate a big array instead of streaming results: 

$result = $db->query('select * from stats');
$array = $result->fetch_all(); // 35M
// or
while ($row = $result->fetch_assoc()) $array[] = $row; // 35M
// or
while ($row = $result->fetch_array()) $array[] = $row; // 44.5M
// process $array ...

// instead of:
while ($row = $result->fetch_assoc()) { // 0.5M
  // process $row
}
Sum with PHP instead of SQL: 

$sum = 0;
foreach ($array as $val) $sum += $val[0]; // 44M, 1.2s

// instead of:
list($sum,) = $db->query('select sum(t1) from stats')->fetch_row(); // 0.2M, 0.1s
Sort with PHP instead of SQL: 

usort($array, function ($a, $b) { return $a[0] > $b[0]; }); // 4.1s
// or
foreach ($array as $key=>$val) $helper[$key] = $val[0];
asort($helper); // 2.2s

// instead of:
$result = $db->query('select * from stats order by c1');
while ($row = $result->fetch_assoc()) { // 1.2s
Let's add a second table: 

$db->query('create table stats2 (c1 int(11) primary key, c2 varchar(255))');
$db->query('begin');
for ($i=50000; $i<51000; $i++) {
  $db->query('insert into stats2 values ('.$i.','.($i*2).')');
}
$db->query('commit');
Join with PHP instead of SQL (join result contains 1000 entries): 

$array = $db->query("select * from stats")->fetch_all();
$array2 = $db->query("select * from stats2")->fetch_all();

foreach ($array as $key=>$val) {
  foreach ($array2 as $key2=>$val2) { // 35.7M, 69s
    if ($val[0] == $val2[0]) // do sth.
  }
}

// instead of:
$result = $db->query('select * from stats a, stats2 b where a.t1=b.t1');
while ($row = $result->fetch_array()) { // 0.5M, 0.015s
Modify arrays without references: 

$array = array();
for ($i=0; $i<1000000; $i++) $array[] = $i*2;

$start = microtime(true);
foreach ($array as &$val) $val++;
echo (memory_get_peak_usage(true)/1048576)."\n"; // 80M (32bit), 200M (64bit)
echo (microtime(true)-$start)."\n"; // 0.14s

$start = microtime(true);
foreach ($array as $key=>$val) $array[$key]++;
echo (memory_get_peak_usage(true)/1048576)."\n"; // 161M (32bit), 399M (64bit)
echo (microtime(true)-$start)."\n"; // 0.64s
more examples coming ...

Scripts running on a 1.4 GHz machine with PHP 5.4.0.