Hadoop (HDFS) is designed to store raw files—it only understands bytes and does not know anything about the format, schema, or structure of the data it holds. However, Hive is a data warehouse system designed to query that data using SQL (HiveQL), meaning it expects structured rows and columns.
Solution-
Serialize = convert an object/data structure into a format that can be stored or transmitted (e.g., bytes, JSON, XML).
Deserialize = convert that stored/transmitted format back into a usable object/data structure.
They're inverse operations of each other.
import json
# Serialize: Python object -> JSON string
data = {"name": "Alice", "age": 30}
serialized = json.dumps(data)
print(serialized) # '{"name": "Alice", "age": 30}'
print(type(serialized)) # <class 'str'>
# Deserialize: JSON string -> Python object
deserialized = json.loads(serialized)
print(deserialized) # {'name': 'Alice', 'age': 30}
print(type(deserialized)) # <class 'dict'>
| Aspect | Serialize | Deserialize |
|---|---|---|
| Use case | Object → String/Bytes | String/Bytes → Object |
| Common formats | JSON, XML, Protobuf, Pickle, MessagePack | Same format, reversed |
UseCase-
Saving to file, sending over network, caching, reading from file, receiving from network, loading cache