NoSQL: Why, When, and How
5 likes6,201 views
The document discusses NoSQL databases and CouchDB. It provides an overview of NoSQL, the different types of NoSQL databases, and when each type would be used. It then focuses on CouchDB, explaining its features like document centric modeling, replication, and fail fast architecture. Examples are given of how to interact with CouchDB using its HTTP API and tools like Resty.
![Time to Relax
(command line)
Apache CouchDB 1.0.2 (LogLevel=info) is starting.
Apache CouchDB has started. Time to relax.
[info] [<0.31.0>] Apache CouchDB has
started on http://127.0.0.1:5984/](https://image.slidesharecdn.com/nosqlwhywhenhow-110609151004-phpapp01/85/NoSQL-Why-When-and-How-44-320.jpg)
![A JSON Document
{
“json”: “key/value pairs”,
“_id” : “some uuid”,
“_rev”: “mvcc key”,
“string keys”:
[1,2,3,”four”,null],
“schema free”:
{“so it’s”:“flexible”}
}](https://image.slidesharecdn.com/nosqlwhywhenhow-110609151004-phpapp01/85/NoSQL-Why-When-and-How-55-320.jpg)
![JSON to PHP Object
$json
=
'{"json":"document","with":["an",
"array"]}';
$j
=
json_decode($json);
//
$j
stdClass
Object
(
[json]
=>
document
[with]
=>
Array
(
[0]
=>
an
[1]
=>
array
)
)
echo
$j‐>with[1];](https://image.slidesharecdn.com/nosqlwhywhenhow-110609151004-phpapp01/85/NoSQL-Why-When-and-How-56-320.jpg)
![JSON to PHP array
$json = '{"json":"document","with":["an",
"array"]}';
$j = json_decode($json, true);
// $j
Array
(
[json] => document
[with] => Array
(
[0] => an
[1] => array
)
)
echo $j[‘with’][1];](https://image.slidesharecdn.com/nosqlwhywhenhow-110609151004-phpapp01/85/NoSQL-Why-When-and-How-57-320.jpg)
![I got tired of having to
pick -> or []
$j
=
new
ArrayObject(
json_decod($json),
ArrayObject::ARRAY_AS_PROPS
);
print_r($j[‘with’][0]);
print_r($j‐>with[0]);
//
“an”
‐‐
same
result!
no
errors!](https://image.slidesharecdn.com/nosqlwhywhenhow-110609151004-phpapp01/85/NoSQL-Why-When-and-How-58-320.jpg)
![code preamble
#!/usr/bin/php
<?php
if ($argc < 2) die('I need a file name.'."n");
// CouchDB config
$user = 'admin';
$password = 'passwd';
// gather and clean EXIF data
$exif = exif_read_data($argv[1]);
unset($exif['MakerNote']);
unset($exif['ComponentsConfiguration']);
unset($exif['JPEGThumbnail']);
unset($exif['TIFFThumbnail']);](https://image.slidesharecdn.com/nosqlwhywhenhow-110609151004-phpapp01/85/NoSQL-Why-When-and-How-79-320.jpg)
![require_once dirname(realpath(__FILE__)).'/../libs/Sag/src/Sag.php';
$sag = new Sag('localhost', 5984);
$sag->setDatabase('pouch');
// PUT the binary attachment first
$sag->setAttachment('original', // the attachment name
file_get_contents($argv[1]), // the file
image_type_to_mime_type(exif_imagetype($argv[1])),
$id = md5_file($argv[1])); // the doc id
// then add the EXIF data
// GET the full doc as we need to add our info to what's there
// atomicity is at a document level in CouchDB
$doc = $sag->get($id)->body;
$doc->exif = $exif;
print_r($sag->put(md5_file($argv[1]), $doc));](https://image.slidesharecdn.com/nosqlwhywhenhow-110609151004-phpapp01/85/NoSQL-Why-When-and-How-80-320.jpg)
NoSQL: Why, When, and How
- 1. NoSQL Why, When, and How starring CouchDB
- 2. aka built rebuilding on hack at contribute to work at
- 4. What is Why NoSQL ^
- 5. NoSQL is... Not Only SQL or SQL I vote for this one
- 6. Origins of the Name Carlo Strozzi ‐ NoSQL app started in 1998. RDBMS sans SQL applied to non‐relational DB’s around 2008‐ish no:sql(east) select fun, profit from real_world where relational=false;
- 7. Origin of the Species non‐relational databases pre‐date relational ones it’s sorta like AJAX been doin’ it for awhile gets a name now it’s cool!!1!
- 8. Types of NoSQL DBs graph (RDF/Semantic Web/triples) key‐value (just what it says) document (k/v + queriability) object (big in the 1980s) multivalue (old tech...like 1960s) NewSQL?
- 9. NewSQL? Recently (April, 2011) coined term by The 451 Group mostly means SQL db’s + better scallability or NoSQL db’s + SQL layers yeah! more keywords!! keeps marketing happy...
- 11. Graph (RDF) pretty heady stuff Web 3.0? Maybe... simple concept, FlockDB (from Twitter) complicated execution (often) AllegroGraph Queried with: Cytoscape SPARQL Java
- 12. Key Value Stores scalable caches generally no query language Project Voldemort get key(s) Hibari return value(s) Big Co need driven
- 13. Document Databases key value + querying Lotus Notes Amazon SimpleDB
- 14. object
- 17. Why
- 18. One Reason
- 19. options
- 22. licensing Apache License 2.0 the favorite AGPLv3 the commercial favorite others include LGPL(v2/3), GPL(v2/3), BSD, MIT, custom commercial or open source ...here there be dragons...
- 23. (un)expected extras at least in CouchDB Built in Web Server or App Server Geospatial bounding box queries n‐master replication binary file storage scales up and down
- 24. When
- 25. Scenarios Scalability caching, sharding Analytics Data Warehousing Ubiquitous/Distributed Data mobile, desktop, server
- 26. Scalability “cache” style DBs data served from RAM (mostly) Membase, Elastic Couchbase, memcached, MongoDB, Casandra horizontal scalability add more servers, not more server
- 28. Ubiquitous/Distributed Data CouchDB server <=> desktop <=> mobile Riak Mobile? go to Erlang Factory in June
- 29. FALE Scenarios power outages data loss failed persistence, no persistence network unavailability special thanks to @coats who runs fale.ca for the FALE stamp
- 31. How with CouchDB
- 32. Time to Relax
- 33. Time to Relax That’s Damien. He built CouchDB.
- 34. That’s the CouchDB gang sign. Time to Relax That’s Damien. He built CouchDB.
- 35. That’s the CouchDB gang sign. Time to Relax Learn it! That’s Damien. He built CouchDB.
- 36. and here’s why
- 37. CouchDB has Super Powers! Schemaless Document centric Replication/Sync Fail Fast Architecture stateless API append only file storage
- 38. Document centric “natural” data model store data like it exists everywhere else: as a document map/reduce vs. sql sorting documents out of a drawer vs. reassembling them from bits of data
- 39. Replication/Sync MVCC‐based transactions versioning...but only meant for transactions safely merge databases documents aren’t compared, only UUID’s & revision ID’s conflicting documents are marked and a winner is picked
- 40. Fail Fast append only database file everything goes on the end of the file querys are cached there too bounce back from errors rather than spin wheels indefinitely
- 42. CouchDB Scaling scales up and down server <=> desktop <=> mobile thanks to n‐master replication BigCouch plugin for sharding HTTP API can be load balanced reverse proxies and caching
- 43. Get CouchDB http://www.couchbase.com/downloads/ couchbase‐server/community http://iriscouch.com/ both have GeoCouch built in! https://cloudant.com/ (BigCouch/sharded) your system’s package installer... http://wiki.apache.org/couchdb/ Installation
- 44. Time to Relax (command line) Apache CouchDB 1.0.2 (LogLevel=info) is starting. Apache CouchDB has started. Time to relax. [info] [<0.31.0>] Apache CouchDB has started on http://127.0.0.1:5984/
- 45. PHP & CouchDB
- 46. HTTP Clients All you really need to get started Does require a better understanding of CouchDB’s API...but that’s A Good Thing! Will improve your HTTP skillz...another Good Thing! requires more work...the part you won’t like but it’s worth it to learn HTTP & REST
- 47. CouchDB Clients CouchDB’s API is just HTTP but...helper libraries can...help: auto (en|de)code JSON handle base64‐ing “inline” attachments manage authentication, cookies, OAuth token exchange caching!!! _changes feed watching
- 48. CouchDB Clients Sag for CouchDB (Apache License 2.0) PHP On Couch (GPLv2 or v3) Beyond here, there be giants... PHPillow (LGPL 3) PHP Object_Freezer (BSD) PHP CouchDB Extension (PHP License 3.0) Doctrine2 CouchDB ODM
- 49. HTTP Clients curl ugh...messy pecl_http lovely (next to curl), but takes some install time, lacks examples Zend_HTTP & PEAR HTTP_Request2 Most major frameworks have their own
- 50. Client Suggestions HTTP pick one that’s flexible (can handle COPY) CouchDB I use Sag currently. Caching, Cookie Auth, nice name. PHP‐on‐Couch seems great, but watch the license (GPL)
- 51. Today’s Stack In PHP: Sag ‐ saggingcouch.com/ For HTTP API Demoing/Testing: Resty ‐ github.com/micha/resty Poster for Firefox code.google.com/p/poster‐ extension/
- 52. Other Handy HTTP Clients HTTPClient for Mac OS X Charles Proxy ($$) http‐twiddle for Emacs Fiddler for Windows Solex for Eclipse
- 53. CouchDB HTTP API we’ll be back to PHP in a bit
- 55. A JSON Document { “json”: “key/value pairs”, “_id” : “some uuid”, “_rev”: “mvcc key”, “string keys”: [1,2,3,”four”,null], “schema free”: {“so it’s”:“flexible”} }
- 56. JSON to PHP Object $json = '{"json":"document","with":["an", "array"]}'; $j = json_decode($json); // $j stdClass Object ( [json] => document [with] => Array ( [0] => an [1] => array ) ) echo $j‐>with[1];
- 57. JSON to PHP array $json = '{"json":"document","with":["an", "array"]}'; $j = json_decode($json, true); // $j Array ( [json] => document [with] => Array ( [0] => an [1] => array ) ) echo $j[‘with’][1];
- 58. I got tired of having to pick -> or [] $j = new ArrayObject( json_decod($json), ArrayObject::ARRAY_AS_PROPS ); print_r($j[‘with’][0]); print_r($j‐>with[0]); // “an” ‐‐ same result! no errors!
- 59. HTTP / REST basics GET read PUT create or update DELETE delete POST bulk operation
- 60. Resty command line RESTful good times
- 66. PUT (failure) on purpose... PUT /pouch/tek ‘{“php”:”tek”}’ {"error":"conflict","reason":"Documen t update conflict."} # we need that _rev value now
- 68. PUT binary attachments PUT /pouch/tek/photo?rev=2‐ fff750985c2c2602e859fe38cd1d347e ‐Q filename.png ‐H “Content‐Type: image/png” {"ok":true,"id":"tek", "rev":"3‐18d519e58b569e43a6fd5e87491f0c4c"} GET /pouch/tek {"_id":"tek", "_rev":"3‐18d519e58b569e43a6fd5e87491f0c4c", "_attachments":{"photo":{"content_type":"image/ png", "revpos":1,"length":18,"stub":true}}}
- 69. A bit about attachments Each attachment to a doc has it’s own URL: /pouch/tek/photo /pouch/tek/schedule.pdf each attachments has it’s own mimetype attachments can be added/updated via their own URLs or inline
- 70. DELETE (failure) on purpose...again DELETE /pouch/tek {"error":"conflict", "reason":"Document update conflict."} # you always have to send a “rev” when changing a doc in any way
- 73. That was just the very basics GET /_stats (server stats) GET /_all_dbs (list all DB’s) GET /_all_docs (list all docs) GET /db/_changes (list recent changes) super powers!
- 74. Map/Reduce Queries searching the file drawer essentially stored queries a.k.a. “Views” written in JavaScript (or Python or Ruby or Erlang or PHP?) similar to array_map()/ array_reduce(), but scalable no ad‐hoc queries
- 75. Pouch put your media in the Couch starring PHP!
- 76. i want to cover schema‐less JSON docs _security Binary Attachments server side rendering stuff Map/Reduce _show Javascript _list several examples URL Rewriting replication _rewrite security vhosts validate_doc_updates
- 77. Pouch is... a Database of JSON docs of file meta‐data with the file attached! a filesystem importer a Web‐based CouchApp for browsing
- 78. 0.1 Files into CouchDB gather EXIF data add attachment PUT to CouchDB let us know how it went
- 79. code preamble #!/usr/bin/php <?php if ($argc < 2) die('I need a file name.'."n"); // CouchDB config $user = 'admin'; $password = 'passwd'; // gather and clean EXIF data $exif = exif_read_data($argv[1]); unset($exif['MakerNote']); unset($exif['ComponentsConfiguration']); unset($exif['JPEGThumbnail']); unset($exif['TIFFThumbnail']);
- 80. require_once dirname(realpath(__FILE__)).'/../libs/Sag/src/Sag.php'; $sag = new Sag('localhost', 5984); $sag->setDatabase('pouch'); // PUT the binary attachment first $sag->setAttachment('original', // the attachment name file_get_contents($argv[1]), // the file image_type_to_mime_type(exif_imagetype($argv[1])), $id = md5_file($argv[1])); // the doc id // then add the EXIF data // GET the full doc as we need to add our info to what's there // atomicity is at a document level in CouchDB $doc = $sag->get($id)->body; $doc->exif = $exif; print_r($sag->put(md5_file($argv[1]), $doc));
- 81. handle updates: GET/PUT or HEAD/PUT or PUT/ error_handle/PUT
- 82. 3 Tier UI for viewing browser <-> php <-> couchdb
- 84. _all_docs for doc list
- 85. loading CouchDB results into JSON and then into template/ PHP output
- 86. Just certain documents ala Map/Reduce
- 87. _view API & query params
- 88. include_docs
- 89. thumbnail creation from PHP async or during upload if we want to wrap CouchDB
- 90. disadvantages of non- async operations
- 91. 2.5 Tier refactoring browser <-> couchdb ^ php
- 92. find photos sans thumbnails and add them
- 93. _changes feed could check this first before hitting the view: _changes is “lighter”
- 94. JS app to replace the PHP-built UI
- 95. PHP cronjob handles thumbs/metadata async
- 96. CouchDB setup on public port
- 97. couchapp script http://couchapp.org
- 99. big advantage: data & app stay together this is huge!
- 100. small disadvantages: dynamic data (thumbs) won’t always be there no real data loss in this case, as they can be re-created later
- 101. _replicate
- 102. Removing Images just DELETE the docs
- 103. Compaction (for space cleanup) _deleted docs _stats (for CouchDB-wide) space usage)
- 105. _security API
- 106. Users/Roles on docs
- 107. CouchDB permissions
- 108. Cookie Authentication (in the CouchApp)
- 109. Basic Authentication (for the cronjob) Sag, HTTP API access, etc.
- 110. OAuth it’s an option, but a whole ‘nother talk
- 111. Replicating with security
- 112. Adding a form for User Data Entru
- 113. - Server-side validation
- 114. validate_doc_updates document validate for additional security
- 115. Making the App “Static” (less AJAX dependent)
- 116. _show “templating” for a single doc
- 117. _list templating for view output
- 118. Migrating existing MySQL- based Gallery data to Pouch
- 119. writing JSON docs pulled from old Gallery
- 120. requesting those docs, PUTing them, PUTing the attachment
- 121. CouchDB as REST API server Building another API for Pouch In addition to (or instead of) the standard CouchDB API Putting that into CouchDB _list ‐ index pages _show ‐ single document page _update ‐ document modification _rewrite ‐ URL Rewriting
- 122. Any questions?
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: \n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: \n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n
- #47: \n
- #48: \n
- #49: \n
- #50: \n
- #51: \n
- #52: \n
- #53: \n
- #54: \n
- #55: \n
- #56: \n
- #57: \n
- #58: \n
- #59: \n
- #60: \n
- #61: \n
- #62: \n
- #63: \n
- #64: \n
- #65: \n
- #66: \n
- #67: \n
- #68: \n
- #69: \n
- #70: \n
- #71: \n
- #72: \n
- #73: \n
- #74: \n
- #75: \n
- #76: \n
- #77: \n
- #78: \n
- #79: \n
- #80: \n
- #81: \n
- #82: \n
- #83: \n
- #84: \n
- #85: \n
- #86: \n
- #87: \n
- #88: \n
- #89: \n
- #90: \n
- #91: \n
- #92: \n
- #93: \n
- #94: \n
- #95: \n
- #96: \n
- #97: \n
- #98: \n
- #99: \n
- #100: \n
- #101: \n
- #102: \n
- #103: \n
- #104: \n
- #105: \n
- #106: \n
- #107: \n
- #108: \n
- #109: \n
- #110: \n
- #111: \n
- #112: \n
- #113: \n
- #114: \n
- #115: \n
- #116: \n
- #117: \n
- #118: \n
- #119: \n
- #120: \n
- #121: \n
- #122: \n
- #123: \n
- #124: \n
- #125: \n
- #126: \n
- #127: \n
- #128: \n