Harden allocation failure handling and size arithmetic

- check malloc/calloc/realloc results in constructors, the JSON and
  OpenStep writers, the XML text-part helpers and base64 decoding, and
  return PLIST_ERR_NO_MEM instead of asserting or dereferencing NULL
- guard size calculations against overflow in the UTF-8/UTF-16BE
  conversions, ptrarray, and bytearray
- make hash_table_insert() report failure instead of silently dropping
  the entry; writers propagate PLIST_ERR_NO_MEM, and dict lookup caches
  are discarded so lookups fall back to a linear search
- don't overwrite an earlier parse error code in the XML parser, and
  report PLIST_ERR_NO_MEM when reading text content fails for lack of memory
- OpenStep parser: check node/data/string allocations and dict/array
  insertion results

Based on #323 by bg0d-droid.

Co-authored-by: bg0d-droid <81825610+bg0d-droid@users.noreply.github.com>

1 files changed, 2 insertions, 1 deletions

diff --git a/src/hashtable.h b/src/hashtable.h
index 514cfec..1b6976d 100644
--- a/src/hashtable.h
+++ b/src/hashtable.h
@@ -43,7 +43,8 @@ typedef struct hashtable_t {
 hashtable_t* hash_table_new(hash_func_t hash_func, compare_func_t compare_func, free_func_t free_func);
 void hash_table_destroy(hashtable_t *ht);
 
-void hash_table_insert(hashtable_t* ht, void *key, void *value);
+/* returns 0 on success, -1 on error (invalid argument or out of memory) */
+int hash_table_insert(hashtable_t* ht, void *key, void *value);
 void* hash_table_lookup(hashtable_t* ht, void *key);
 void hash_table_remove(hashtable_t* ht, void *key);