Feed aggregator

After a major cleanup of the structure and logic due to a handful of issues, v4.1 of the Project Importer is now available!

You can download the latest version here:
https://sourceforge.net/projects/web2project-mod/files/Project-Importer/

The purpose of the Project Importer is to allow users to import their project plans from other systems. Currently Microsoft Project and WBS Gantt Chart Pro are supported.
The significant changes in this release include:

• Fixed a major issue where dependencies weren't being imported properly or sometimes at all.
• Added a tweak to the MS Project processor to make sure the project title is retrieved properly.
• Fixed an issue where the Project Start Date was not set properly in the MS Project import
• Tweaked the import process so that if there's no Company match, it default's to the current User's Company. This resolves the issue where a Project could accidentally get assigned to no Company and therefore disappear due to permissions.
• Changed the licensing to align with the coming web2project license shift.

There is a goal to support additional file type imports. At present, CSV - as an Excel export - is one option but is the most difficult due to a lack of formal/official column/field structure that a user could create. If anyone has any suggestions, please feel free to share them in our forums.

The Project Importer can be downloaded here: https://sourceforge.net/projects/web2project-mod/files/Project-Importer/

We recently rolled out a new version control process, which we feel will better support our increasing project complexity.

Previously, our live sites used trunk, and our work took place on a single dev branch. This was problematic because it was sometimes difficult to separate the tested-and-ready-for-live code from the code still in development.

Now, we use multiple branches (one for each major enhancement and/or batch of bugfixes), and tag any finished changes. Tagging takes the guesswork out of figuring out what code is ready for live. When we need to cutover some changes, it's now as simple as switching to the newest tag.

Initial Buildout

1. Checkout a copy of trunk into your local working directory:
   
   1. svn checkout svn+ssh://path/to/project/trunk project
2. Develop. Commit often. When the functionality is complete, create a branch:
   1. svn ci -m "Committing trunk changes"
   2. svn copy svn+ssh://path/to/project/trunk svn+ssh://path/to/project/branches/1-0-x
3. Switch to the new branch:
   1. svn switch svn+ssh://path/to/project/branches/1-0-x
4. Test. When finished, tag your new release:
   1. svn copy svn+ssh://path/to/project/branches/1-0-x svn+ssh://path/to/project/tags/1-0-0
5. Merge the branch back into trunk:
   1. svn switch svn+ssh://path/to/project/trunk
   2. svn merge ^/trunk@HEAD ^/branches/1-0-x@HEAD
   3. Resolve any conflicts. Merging can sometimes be a tricky and time-consuming process. When in doubt, ask for help. There are also several GUI tools to make merging easier. Beyond Compare 3, WinMerge, and TortoiseSVN are some choices for Windows.
6. Commit the merge
   
   1. svn commit -m "Merged branch 1-0-x back into trunk"

Bugfixes

1.  Switch to the latest branch number:
   
   1. svn switch http://path/to/project/branches/1-3-x
2.  Develop, then test. When finished, tag your release with the next minor version:
   
   1. svn copy svn+ssh://path/to/project/branches/1-3-x svn+ssh://path/to/project/tags/1-3-2
3. Repeat steps 5-6 of Initial Buildout.

 New Features

1. For major, long-running development that needs to happen simultaneously with smaller feature development, consider adding a new feature branch. Otherwise, work on trunk.
2. Repeat steps 1-6 of Initial Buildout, remembering to update major version numbers (e.g. branches/1-1-x).
3. As a good practice, make sure to do svn update before doing any commits.

 Updating the Live Site

1. Switch to the latest tag:
   1. svn switch svn+ssh://path/to/project/tags/1-1-0

 

July was a month of talks and travel, including speaking at OSCON and user group talks to DCPHP and PDXPHP. For those who saw the “Micro Optimize This!” talk, you can download the slides here. For those looking for the “Five Tips To Make Good Object Oriented Code Better” slides, those are available here.

So I created a poorly constructed 'SELECT' query in a stored procedure and I was up late last night trying to figure out how I could best optimize it. After sitting in my 94 degree basement for a few hours, I thought of using hashed values via CONCAT and MD5 functions to create an index of multiple columns to improve 'SELECT' queries using 'JOINS.' The performance increase was instant after I made the necessary adjustments. To save others the time and headache, I'm going to share with you my solution and the resulting performance improvements.

So, I am dealing with two tables called "meetup_events" and "location" containing 1,000+ and 290,000+ records, respectively. Both tables contain information for:

• location name
• location address
• location city
• location state
• location zip
• location latitude
• location longitude
• location phone

meetup_events table Field Type Null Key Default Extra id int(11) NO PRI NULL   name varchar(100) NO   NULL   description text NO   NULL   time timestamp NO   CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP group_id varchar(100) NO   NULL   group_name varchar(100) NO   NULL   group_photo_url varchar(200) NO   NULL   fee varchar(100) NO   NULL   how_to_find text NO   NULL   organizer_name varchar(100) NO   NULL   nid int(11) NO   NULL   status int(11) NO   0   venue_name varchar(100) NO   NULL   venue_address varchar(100) NO   NULL   venue_city varchar(100) NO   NULL   venue_state char(5) NO   NULL   venue_zip char(5) NO   NULL   venue_phone varchar(15) NO   NULL   venue_lon decimal(10,6) NO   NULL   venue_lat decimal(10,6) NO   NULL  

 

location table Field Type Null Key Default Extra lid int(10) unsigned NO PRI NULL auto_increment name varchar(255) NO       street varchar(255) NO       additional varchar(255) NO       city varchar(255) NO       province varchar(16) NO       postal_code varchar(16) NO       country char(2) NO       latitude decimal(10,6) NO   0.000000   longitude decimal(10,6) NO   0.000000   source tinyint(4) NO   0   is_primary tinyint(4) NO   0  

 

The data fields that I need to extract include locations from the meetup_events table that are not in the location table. The query that I was using originally is:

SELECT DISTINCT
`me`.`venue_name`,`me`.`venue_address`, `me`.`venue_city`,`me`.`venue_state`,
`me`.`venue_zip`,`me`.`venue_phone`, `me`.`venue_lon`,`me`.`venue_lat`
FROM `meetup_events` `me` LEFT JOIN `location` `l`
ON( `me`.`venue_address` = `l`.`street`
    AND `me`.`venue_state` = `l`.`province`
    AND `me`.`venue_city` = `l`.`city`)
WHERE `me`.`venue_lat` <> 0 AND isnull(`l`.`name`)

Unfortunately, there is no immediate way of identifying a relationship between the location associated with each meetup_event and a record in the location table that uses an index because the meetup_events table contains data that is imported from a third-party source. This is where I took a wrong turn and JOIN'ed the tables using the address, state, and city fields.

...
ON( `me`.`venue_address` = `l`.`street`
    AND `me`.`venue_state` = `l`.`province`
    AND `me`.`venue_city` = `l`.`city`)
...

This didn't pose an issue when the number of records in the location table was smaller, but as the table grew it failed to scale and resulted in ERROR: query execution was interrupted. Not fun!

(click to see full image)

So after thinking about the minimal data that I would need to constuct a unique identifer for a given location, I decided to turn my attention to the location/venue name, latitude, and longitude data. This will take care of potential duplicates such as locations with mutliple venues like malls, multi-storied buildings, etc.

Now that I have figured out that aspect, I must store the data in a format that is optimial for the MySQL to quickly read and index. Without thought, I turned to the CONCAT() function to provide a single field to store the data and return a scalar value to store. 

CONCAT(name, latitude, longitude);

To ensure that the data remains relatively short and unique, the MD5() function comes into play which will result in a hashed value of the concatenated value returned by CONCAT(name, latitude, longitude).

MD5(CONCAT(name, latitude, longitude));

Example: SELECT MD5(CONCAT('The Park at 14th', 38.879456, -76.985059));

MD5(CONCAT('The Park at 14th',38.879456,-76.985059))

e3dd1caa1253f2f5def9caadc3e474b7

 

Using this means of generating a unique value, both tables (meetup_events and location) are given a new field called, hashed_index of type varchar(100) with an INDEX.

meetup_events table (with new field) Field Type Null Key Default Extra id int(11) NO PRI NULL   name varchar(100) NO   NULL   description text NO   NULL   time timestamp NO   CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP group_id varchar(100) NO   NULL   group_name varchar(100) NO   NULL   group_photo_url varchar(200) NO   NULL   fee varchar(100) NO   NULL   how_to_find text NO   NULL   organizer_name varchar(100) NO   NULL   nid int(11) NO   NULL   status int(11) NO   0   venue_name varchar(100) NO   NULL   venue_address varchar(100) NO   NULL   venue_city varchar(100) NO   NULL   venue_state char(5) NO   NULL   venue_zip char(5) NO   NULL   venue_phone varchar(15) NO   NULL   venue_lon decimal(10,6) NO   NULL   venue_lat decimal(10,6) NO   NULL   hashed_index varchar(50) NO MUL NULL  

 

location table (with new field) Field Type Null Key Default Extra lid int(10) unsigned NO PRI NULL auto_increment name varchar(255) NO       street varchar(255) NO       additional varchar(255) NO       city varchar(255) NO       province varchar(16) NO       postal_code varchar(16) NO       country char(2) NO       latitude decimal(10,6) NO   0.000000   longitude decimal(10,6) NO   0.000000   source tinyint(4) NO   0   is_primary tinyint(4) NO   0   hashed_index varchar(50) NO MUL NULL  

 

Here is the query rewritten and optimized:

SELECT DISTINCT
`me`.`venue_name`,`me`.`venue_address`,
`me`.`venue_city`,`me`.`venue_state`,
`me`.`venue_zip`,`me`.`venue_phone`,
`me`.`venue_lon`,`me`.`venue_lat`
FROM `meetup_events` `me` LEFT JOIN `location` `l`
USING( hashed_index)
WHERE `me`.`venue_lat` <> 0 AND isnull(`l`.`name`);

(click to see full image)

If you have any multiple column indexes within any of your tables, try indexing a hashed version of those fields concatenated, and then test to see if this results in a performance increase. Good luck!

For those of you just tuning into the WordPress/Thesis battle, here is the current - as of 19 July 2010 - state of things:

• At some point recently, the servers of Chris Pearson - owner and distributor of the Thesis theme for WordPress - had a vulnerability on his server and the latest releases of Thesis were compromised.
• When that came to light, Matt Mullenweg - head of the WordPress project, founder of Automattic, and founder of the WordPress Foundation - has previous stated that "WordPress Themes are GPL too!" sent a simple Tweet: "This is what happens when non-coders think they can code."
• In a matter of moments, that was being reTweeted, argued, discussed, blame was being spread around, and things started getting heated.
• Andrew Warner had the insight to get both Chris and Matt on the phone for a Mixergy interview. Chris came off poorly (at best) when he "explained" that the GPL didn't apply to Thesis and further claimed that he was one of the three most important people in WordPress. Things exploded from there.
• Matt raised the stakes by making a public offer to buy any existing Thesis customer a new premium theme from somewhere else. And then started tweeting individual Thesis customers making the offer.
• And igniting things further, Andrew Nacin - another core WordPress developer - and a few others (Andy Peatling, Drew Blas) started digging and found snippets of GPL'd WordPress code in Thesis.
• Next, a commenter on Matt's update post (titled: "Syn-Thesis 1 and Chris Pearson") who happened to be a former Thesis developer admitted to copy/pasting things from WordPress because he didn't understand the license.
• Finally, Chris says that the offending code has or will be removed but Mark Jaquith (and the Drupal and Joomla communities) wrote "Why WordPress Themes are Derivative of WordPress" so offending code or not, it doesn't matter.

So what does this all mean to the community?

Good question... As my partner Marco Tabini notes in "WordPress, the GPL, and cherries on top" everyone has an opinion on what the GPL means and what its ramifications are but since there's no legal precedent, it's just a best guess... and there has yet to be a precedent to solidify an interpretation. In the meantime, the most common interpretation is based on the Software Freedom Law Center's opinion and the GNU FAQ. Here's the problem with that:

The FAQ is not part of the license and not distributed with it. It is stored on a website without version control or an audit trail on who might have modified it when. By all accounts, it is less reliable than Wikipedia because even errors can't be fixed.

Feel free to cite the FAQ all you want.. no one ever explicitly or implicitly agreed to that interpretation.

If you combine this with Matt and Mark's opinion that themes are GPL because they're dependent on WordPress code and datastructures to have meaning, what are the ramifications of this interpretation?

• What is the threshold where including code crosses from "Fair Use" into "now this must be GPL"? While there are lots of ways to do things in PHP, if there are Best Practices, we'll all tend to reach similar results.

• For client work, if copyright isn't transfered with the delivery of the code, isn't this distribution? And if so, doesn't the client work have to be GPL? Or if you transfer the copyright at delivery, what about the demo versions? Did you send them copies of the code before you transfered copyright?

• If the client work does have to be GPL, do you have to make it available to anyone who asks for it? Since the GPL doesn't requite a "public disclosure", it's possible no one would know to ask.. but what about employees of either group? If it's "privately" GPL, would an employee be within their rights to take and use a copy?

• What does this do to Non-Disclosure Agreements? If the code must be GPL and I have to provide a copy to whoever might ask, how can I agree to protect the secrets of my customers?

• What about a book that covers GPL code and datastructures? What if the book goes as far as including snippets of WordPress code itself? While the book may be able to physically exist and function independent from the code, it doesn't have any meaning without the core system. Should Lisa Sabin-Wilson GPL her book "WordPress for Dummies"? Does Aaron Brazell have to GPL his "WordPress Bible"?

• A given WordPress post consists of a title, body, and a category or some tags. It's pretty trivial to turn that into a post on the page without using WordPress at all. But what about a complex data structure that only has meaning once it is transformed by the core code?

That last question is the point of the discussion we've had within the web2project team recently.

If you're not familiar with web2project - or its parent dotProject - the data in the database doesn't mean much by itself. The core code must retrieve the data and process it to express a project plan in a meaningful way. It does the same to files uploaded, tasklogs stored, and a number of other things within the system.

Therefore, since the data is wholly dependent on the core web2project code, does the data itself have to be GPL?

And that's why a few weeks back, we started the process of changing licenses from GPL back to BSD as dotProject originally was. I'll go into more detail on the process and due dilligence involved, but know that we've been on this for quite a while and are working to have it fully resolved before the next web2project release.

Disclosures: I don't have an interest in Thesis as I've never used it and don't know Chris Pearson at all. I use Automattic's Akismet for blocking spam on a number of Drupal sites. Finally, I am using WordPress on the web2project site relaunch and was a reviewer in Aaron Brazell's WordPress Bible and wrote the foreword. My biggest concern in all of this are the larger implications on web2project and GPL projects in general.

Several battles have been brewing from within the WordPress community.

It's WordPress, not Wordpress dangit!

A couple weeks ago, a snippet of code got sneaked into WordPress 3.0. The function, capital_P_dangit, is a filter that replaces "Wordpress" with "WordPress" throughout one's site.

While this doesn't seem like a big deal, this change infuriated some users. They claimed that WordPress does not own a user's content, and therefore, such a move (without first explicitly asking to do so) infringes on the user's expressive freedoms.

The filter itself doesn't bother me personally. What bothers me is the lack of review. This change produced a rarely-occuring permalink bug in WordPress 3.0. This "feature" was added with barely any discussion, and hopefully isn't an indication of things to come.

Themes and Plugins Must be GPL

WordPress is licensed under the GPL (v2). It's been clearly stated that all WordPress plugins and themes should be licensed under the GPL as well.

As written on Perpetual Beta, "Anyone who gets a copy of a premium theme then has the right to freely distribute it or modify it virtually without restriction (expect, of course, those restrictions found in the GPL itself)".

The popular paid theme, Thesis, does not adhere to the GPL license. The authors refuse to.

The GPL (version 2)

The GPL ensures that software is freely redistributable, but not necessarily without a price. You've probably heard the phrase "free as in free speech, not as in free beer." GPL software is meant to protect its users (and the community), not the authors. Users have four essential freedoms:

• The freedom to run the program, for any purpose (freedom 0).
• The freedom to study how the program works, and change it to make it do what you wish (freedom 1). Access to the source code is a precondition for this.
• The freedom to redistribute copies so you can help your neighbor (freedom 2).
• The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

Thesis is Special, Right?

The authors, particularly Chris Pearson, fear that switching to GPL will de-value their product. It'll allow for knock-off themes that are strikingly similar to Thesis — but for a fraction of the cost.

Under the GPL license, free distribution is encouraged -- as long as the product (or derivative) remains under the GPL.

The other argument is that although Thesis requires WordPress to function, the vast majority of the theme's code is totally original. In that respect, some argue that it may not fall into the "derivative work" category, which mandates the GPL stamp.

Chris claimed during a live call that WordPress cannot lawfully force Thesis to adopt the GPL. He coined the GPL as a "flimsy and unenforceable license".

No, says WordPress

The creator of WordPress, Matt Mullenweg, argued that licensing Thesis under the GPL was "the right thing to do." He argued that commercial GPL themes (and plugins) can still be profited from, and there are other effective models at earning a steady income (such as through offering services and/or paid support).

Most notably, those supporting Matt's stance argue, "If you don't agree with the GPL, then don't develop for WordPress."

Although both sides have decent points, I was a little disappointed by all the smearing via Twitter. At one point, Matt essentially told followers not to use Thesis, and (in response to a Thesis vulnurability claim), "This is what happens when non-coders think they can code." This isn't saying that Chris wasn't without flaws, as his arrogance shined during the live call: "I am one of the three most important people in WordPress."

Conclusion

This is an ongoing battle, with both sides clinging to their interpretations of the GPL license. I don't see either side giving up very easily, so this dispute could likely find its way into court. I'm excited to see how this unfolds, as a decision will affect not just WordPress, but many other open platforms (Drupal, Joomla, etc.) as well.

What are your thoughts?

There has been a lot of (virtual) ink spilled over HipHop, Facebook's new PHP-to-C compiler; but a similar approach has flown a little more under the radar.  I'm talking about Quercus, a PHP interpreter/compiler written in Java by Caucho, the makers of the Resin app server.

Quercus is a servlet that can be deployed in any J2SE/J2EE servlet container — Resin, Tomcat, JBoss, WebSphere, etc. — and can run PHP applications deployed in the same webapp. Of course the question is why would you use Quercus when you can use Zend's PHP interpreter?

Unlike HipHop where the biggest reason to use it is performance, the case for Quercus is slightly more nuanced. Quercus has more capabilities than HipHop out of the box; for instance you can drop Drupal, WordPress, Joomla, MediaWiki, Gallery2 and more in Quercus and they run. Need to use eval (I'm looking at you, Drupal)? No problem.

Similarly, unlike HipHop which contains a built-in web server, Quercus is a servlet which provides you more  options on how you run it.  And because we're running on Java we have access to all the tasty goodies that people have written for it. Need asynchronous messaging? Love JMS?  Multi-node caching? Terracotta and Ehcache rocks — interestingly enough someone already did this with Drupal. The list goes on and on...Apache Axis is great for web services, GraniteDS does almost as much as Adobe LifeCycle to connect with Flex.

When you look at the documentation for Quercus, they talk about performance being at least comparable to Apache/PHP, and they're mostly correct. Deploying a fairly intensive community Drupal site (OG, Spaces, lots of Views) on Resin 4 and testing with Apache Bench (AB) on a workstation puts Resin running only slightly slower than Apache, at least up to 50 concurrent users.

There is some room to grow on the Resin side as it was running pure Java, no JNI, and anecdotal testing indicates that this provides some 10-12 percent performance benefit. So out of the box this would put Resin processing requests at about the same speed as Apache.

We were testing the community version of Resin and Quercus, the Professional version has the ability to compile PHP to Java using the same sort of static analysis techniques used by HipHop and, at least according to benchmarks published by Caucho (PDF), show an increase in performace of Drupal, WordPress, and other common PHP applications, ranging from ~17 percent in phpBB to almost 110 percent for WordPress. 

So, what's not to love, right? We can take existing PHP applications, move them over painlessly and get both increased performance and capability.

Sadly, things are not that clear-cut. Going back to the example below of the 50 concurrent connections. While performance was roughly equivalent, the effect on system resources was not, Resin used considerably more memory than the comparable test on Apache. Another issue is lack of sophisticated URL rewriting on some servlet containers. Resin supports enough of the functionality of mod_rewrite to make Drupal work, but others do not. Getting Drupal clean URLs in Tomcat, for example, is difficult. Other uses of mod_rewrite may not work at all without significant effort.

And perhaps the biggest downside for Quercus is inherent in what it is. It's Java. For organizations that have made a significant investment in PHP the code is not all there is. You have systems tuned for the LAMP stack. You have system administrators that are knowledgable and experienced in maintaining those systems. You build solutions that use PHP, and while the additional functionality of Java is nice, what do you do with it?  But like HipHop, the very existance of Quercus is encouraging.  While I don't have a project right now targeting Quercus, and I may never, the fact that companies like Facebook and Caucho are innovating and adding new options, new features makes this an exciting time.

Update: To be fair, a great deal of inspiration for this post came from Ed "Funkatron" Finkler and his session at phpWorks 2008 Picachu pitch-at-ya peek-at-you lightning talk called "Users are Assholes" and later by Matthew Weier O'Phinney's lightning talk called "How to get kicked off my Project." Thanks for the fodder guys!

On all sides of software development, there are annoyances. Some of them go beyond annoyances and into pet peeves. Some of them go beyond pet peeves and incite open war within projects and communities. After witnessing a number of these rants - from developers and non-developers - and even sharing a few myself, I thought maybe I could make the situation better.

Therefore, I present here for comment:

The Ten Commandments of Open Source Software.

0. Thou shalt not ask questions without basic research including documentation and the Google.

I. Thou shalt realize that most developers work on passion and freetime. Thou are not paying them.

II. Thou shalt not criticize code and architectures before seeking understanding.

III. Thou shalt not use the phrase "this should be simple" unless thou has confirmed it as such.

IV. Thou shalt apply patches and updates in a timely manner. Further, thou shalt have a vague idea of when the next release is due-eth.

V. Honor thy developers and designers.

VI. Thou shalt include specific error messages and screenshots when thou hast problems. Thou stating "thy software sucks" is not helpful and angers the Funkatron.

VII. Thou shalt not murder.*

VIII. Thou shalt not copy without respecting both the license and the terms within.

IX. Thou shalt not covet the features and functionality in thy neighbor's software.

Did I miss any?

* No, I'm not kidding.

As of 29 June 2010, web2project v2.0 is officially released! You can download it from SourceForge now.

Although this release had lots of bug fixes, the primary focus was on a few specific new features and major pieces of functionality. You can read the full v2.0 Release Notes, but in my opinion, the six most important items are:

• User-based Timezones: Everywhere a time is used or displayed within the system, it's now stored in GMT/UTC and presented in the user's local timezone. If you set a meeting for 5pm America/New_York, someone with America/Chicago timezone will automatically see it at 4pm. If you have team members spread across timezones, this is vital. Lots of thanks to Derick Rethans - the master of all things time-related in PHP - on his numerous presentations on DateTime and Timezones. His information made it possible.
• Unit Tests have come a long way since the v1.0 release. We had zero at that time but Trevor Morse - founder of the Halifax, NS PHP Group and core web2project member - has led the way to 240+ tests focused on the Tasks and Projects classes.
• Subprojects are now Useful: Previously you could denote one project as a subproject as another but it didn't really do anything, it was just presented a bit differently. With this release, when you assign a project as a subproject, now it creates a token task within the parent project. This token task takes on the start/end dates, duration, hours worked, and percent complete of the subproject. As the subproject updates, the token task updates. Even more usefully, you can use the Token Task as a dependency for other tasks in the parent project. That all sounds complicated, so just try it out.
• The class structure has been completely restructured: While this won't be relevant to 99% of our users, it makes it much easier to add standalone frameworks - like the Zend Framework or whatever - to the system for other functionality. On a related point, web2project now supports the naming conventions put forth in the Framework Interoperability Group. This will also allow easier third-party authenticators for systems like Drupal, Joomla, or WordPress.
• Audit Logs - We cleaned up the core system objects to provide historical logging of all CRUD operations. Any Add On modules that use the core objects will get this functionality by default.
• Added an 'update checker' - This is a regular script which runs to notify the System Administrator that a new release is available and collects basic system information. This was modeled after Drupal's update functionality. No sensitive information is collected and this can be opted out of via the System Configuration.

A number of community members stepped up and did a great job in reporting issues, helping test fixes and release candidates, and generally being insightful. Special thanks goes to opto, adolfo, zbyszek, and egemme. Without them, v2.0 would not be as solid, useful, and generally as bug-free as it is.

In summary, we closed about 79 items with ranging from 15 crash-level issues to 42 minor bugs. Once again, those are just the formally reported issues. If you want to explore everything of interest, check out the web2project v2.0 Release Notes on our wiki. And of course, if you're looking for ways to share your code more easily, you should check out our web2project git repository.

You can download web2project v2.0 from SourceForge now.

* And yes, I always wait a few days to announce the releases in case we have to make a patch release. ;)

One frequent request we get from clients is to integrate their Drupal web site user profiles with a Customer Relationship Management (CRM) system so that their users' information can be used for email messaging and other purposes.

We've worked with a variety of CRMs including CiviCRM, SalesForce, Avectra netFORUM, Convio, and Democracy In Action's Salsa.

Salsa has a focus on advocacy activities including creating and signing petitions, generating letters to the editor, and fundraising. so it's a natural fit for a lot of our work with nonprofits, NGOs, and other public policy organizations.

When built EarthDay.org, we created a social networking site for users to register and commit to the cause, and at the end everything had to flow into Salsa to serve as one master list of events, petitions, and individuals for the Earth Day Network. 

In order to make the process as flexible as possible, we based our work on the Drupal Rules module, adding actions to create Supporters, Events, Petitions, Campaigns and Groups from nodes in Drupal.

Below is a screencast showing some of the potential. I take you through the process of creating a new user with a Content Profile node and creating a Supporter in Salsa. Then, I create an Event in Salsa from one in Drupal and have a user register for that Event. The result is that having their Supporter record is correctly associated with the Event in Salsa.

This new module is available on drupal.org at www.drupal.org/project/salsa_rules. I hope you find it useful to your work.

In my last post, I glossed over MongoDB from a very high level. While it's useful to know how MongoDB works, I didn't cover exactly how to use it.

There are a growing number of NoSQL systems, and one of MongoDB's greatest strengths is its ease-of-use. It takes a couple of minutes to install the server, and commands are given using the familiar JSON syntax.

Setting up MongoDB

I'm running Windows, but installing on a Mac or *nix system shouldn't be much different. After downloading the appropriate package, copy the mongdb folder into c:/data/db/. Then, using your shell (cmd prompt), browse to c:/data/db/mongodb/bin and execute the server executable, mongod.exe:

Voila! The MongoDB server is now up and running. Let's fire up a separate window, this time for an instance of the MongoDB client. The client is mongo.exe, also in the bin directory.

Setting up a User Directory

We want to create a simple user directory, and want to store the following information about each user:

name age languages

Unlike with traditional SQL systems, we don't have to create a database schema. You simply create a JSON object containing the data you want, and MongoDB saves it as a document. By default, MongoDB will use the test database. Let's change that to a new database called users:

use users;

I know, it was pretty intense. Once you've wiped the sweat from your brow, let's add some data to our new database:

var data = [ {name: "John Doe", age: 28, languages: ["English", "Arabic"]}, {name: "Mike Smith", age: 35, languages: ["English"]}, {name: "Sally Mae", age: 41, languages: ["Italian", "French", "Arabic"]} ];

 To add these people to our database, we run the following command:

db.users.save(data); Getting all users

 To find all items in the collection, you'd run db.users.find().

This returns our users. If there are more than 10 users, you'll need to enter the "it" command to iterate through the rest of the results. Fortunately, the MongoDB PHP extension makes iterating a lot less tedius.

Getting specific users

You're able to enter specific parameters into the find() function to retrieve filtered results.

To find "John Doe":

db.users.find({name: "John Doe"});

To find anyone 41 years old:

db.users.find({age: 41});

To find anyone who speaks Arabic:

db.users.find({languages: "Arabic"});

MongoDB also supports comparison operators (<, >, <=, >=, etc), but the format takes a bit to get used to. For example, let's find users between 30 and 40 years old:

db.users.find({age: {'$gte': 30}, {'$lte': 40}});

NOTE: "Less than" is '$lt', whereas "Less than or equal" is '$lte'.

Snafus

When using MongoDB, you'll run into difficulties that were non-issues in MySQL. One example is using "OR" when building queries. In MySQL, it's quick and painless to use OR when finding records. Let's find users <= 30 or >= 40 years old:

MySQL

SELECT * FROM users WHERE age <= 30 OR age >= 40;

MongoDB

The "OR" operator was just added to a nightly build of MongoDB, but is unavailable in the current stable version. It's still possible, but it requires a custom '$where' function:

db.users.find({'$where': function() { return this.age <= 30 || this.age >= 40; }}); Conclusion

MongoDB seems to be a pretty powerful tool for document storage. It's fairly robust, extremely fast (from what I can tell), and easy to use.

I cannot stress enough that MongoDB (or any NoSQL solution) is not a substitute for all things SQL. If your application has transactional or relational requirements, you're better off sticking with SQL.

MongoDB is one of the forerunners in the NoSQL movement, an effort to promote non-relational, schema-free data stores. It lacks any table JOINs, which avoids performance bottlenecks seen with traditional SQL servers. 

Why NoSQL?

NoSQL servers are not meant to replace traditional SQL servers. They are meant to handle problems without heavy transactional requirements, but with the potential to massively scale if needed. They work well for providing quick access to large number of documents, serving pages on high-traffic web sites, or delivering streaming media.

Document Storage: Database > Collection > Document

MongoDB is a document database. Documents are stored in collections, and collections are in turn stored in a database. A collection is similar to a table in MySQL (it's a named group of documents), but a collection lacks any schema. 

Documents

A document contains the actual data, and is stored as a binary JSON object (called BSON). A single document has a storage limit of 4 MB. Queries are expressed as JSON-style objects, making it pretty painless to save and retrieve data:

var data = {first_name: "John", last_name: "Doe", lottery: [42, 16, 29]};
db.users.save(data); 

Files can be stored by MongoDB by using the GridFS specification. Since a document can be a maximum of 4 MB, GridFS works by splitting a large file into (usually 256k) chunks before storing it into a files collection. Unfortunately, MongoDB doesn't do any sort of automatic cleanup in the event of a processing error (your collection would be stuck with fragments of the corrupted file). 

Horizontal Scalability

MongoDB uses (auto) sharding, not replication, as a way of achieving high scalability. Sharding essentially involves "breaking your database down into smaller chunks called shards and spreading those across a number of distributed servers. With Mongo, we tend to think of replication as a way to gain reliability/failover rather than scalability. (See this article for more.)

Performance

• There's a client driver per language
  • CouchDB uses REST
• Documents have a maximum size of 4MB
  • This is not changeable
• Memory-mapped files for data storage
  • This means that data is limited to around 2GB on 32-bit systems
  • MongoDB stores as much data in RAM as possible
• Update-in-place (instead of MVCC, as in CouchDB)
• Written in C++ 

Stay tuned for my next post, which will cover the creation of a simple web app using Mongo and the PHP driver.

It's been a week since tekX completed and there are a number of other tekX writeups to read but I thought I'd share one last one from a different point of view. To add some context, I'm not the guy that signs the contracts, approves the expenses, schedules the sessions, arranges the speakers, or anything useful. Due to other project commitments, I basically served as backup for some of those things and then as social chair for the evening events.

First off, I'm proud of our first time tekX speakers. Easily half of the speakers had never spoken at a tek before, off the top of my head that includes - Bill Karwin, David Strauss, Jason Austin, Josh Holmes, Kanwalijeet Singla, Kristina Chodorow, Matt Schmidt, Matt Turland, Ryan Stewart, Sumit Chawla, and myself. Since tek is the Community Conference and it tends to be incredibly technical, it takes a different mindset than many of the other conferences. Even better, a few of those people were first or second time conference speakers period.

Next, based on the feedback from Joind.in, the sessions themselves were consistently solid and "good" at minimum. The speakers did a good job of choosing topics, content, and styles that sparked the interest of attendees. Even better, I regularly heard "I never thought of it that way" in the hallways and at meals. That gets me excited. When someone thinks of something differently or new because of an idea of perspective that a speaker introduced, things can change. They can improve what they're doing. Their organization can improve. The community can improve. Everyone wins.

Next, there were a few sessions that stood out as particularly relevant and interesting. Josh Holmes' opening keynote on The Lost Art of Simplicity. Although I wasn't there, Kristina Chodorow's three hour tutorial on MongoDB topics was received incredibly well and generated lots of discussion. Ryan Stewart's session on the last day included an overview of Data Visualization gave a good overview of tools to build graphs and charts. And despite being an Adobe guy, he did talk about Canvas and non-Adobe options. Finally, I've heard great things about Lorna Jan Michell's session "Open Source your Career".

Next, we had some great sponsors. In alphabetical order: Adobe, Microsoft, SQLyog, and Zend each contributed good things ranging from happy hour drinks to toys to presence to heavy involvement and planning support.

Next, the internet stayed up. No, seriously... it did. We ran out of IP addresses a couple times and it dropped to 1-5kbps at least a couple times, but it was still up. I'm not sure what was so different this year, but it worked.

Finally, once again, the people were amazing. The energy, the excitement, the comradery, the chaos, and the passion was evident. Sure, it takes some people some time to warm up, but once people start connecting, it's hard to avoid it. After tek last year, Trevor Morse (of web2project fame) started the Halifax, Nova Scotia PHP Developers' Group, but this year, this exploded. I moderated a Community User Group Panel with Michelangelo van Dam, Rafael Dohms, Lorna Jane Mitchell, and Ben Ramsey... and apparently, it pushed a few people over the edge to start their own groups.

And with that, I'd like to announce:

Chance Garcia started a local Bloomington, Indiana PHP Developers' Group.

Jeremy Kendall kicked off a the Memphis, Tennessee PHP Developers' Group.

Chris Tankersly started a semi-regional Northwest Ohio PHP Developers' Group.

These guys are taking a step out into their communities with the goal of connecting local developers, employers, recruiters, and everyone else with the goal of making their own skills - and the skills of the people around them - better. If you're in either area, join and help them. If you know people in those areas, pass the word. Even if you're not local but want to lend support, join their lists and contribute.

Even the biggest groups started with a single person with an idea.

During my Live Coding sessions at TEK·X* last week, one of the questions that came up repeatedly was:

Events... what are these Event things and why does Flex work like that?

When you initially dive into the world of Flex development**, most PHP'ers will quickly notice something weird. We're out of the world of Request/Response that we know and understand and into an odd world of Events, Listeners, and Publishers/Subscribers where things just don't play well together.

The familiar Request/Response process is similar to going through checkout to the grocery store. The cashier is idle until a new customer appears. The customer makes their requests, the cashier transfers a specified amount from one account to another, and the customer leaves with their results. As more customers appear and the line gets longer, the stores scales horizontally by adding more cashiers. It all scales quickly and easily with minimal effort because grocery store cashiers are a shared-nothing architecture.

Alternatively, the Publisher/Subscriber (or Observer) pattern is similar to waiting at the airport. When you're waiting for a flight, you sit near your gate and wait for an announcement. In the meantime, you might grab a cup of coffee, use the restroom, check email, or make phone calls. The people around you do the same. As announcements are made, some people board their flights or swear at gate agents but regardless until your announcement, you don't care. But once you you hear your announcement, everything changes. You finish whatever you're doing and take action.. along with anyone else listening for the same flight.

This is the core of Event Driven Programming.

Flex** works the same way. As you create elements and functions/methods, they listen for different Events to "fire". Once the Event occurs, all the Listeners attempt their task. The vast majority of these Listeners will not interact with one another but in some cases, Events will fire other Events.

A common task - like Uploading a File - will consist of a number of these Events:

• First, we have to select the file to upload. This could be done via a drag and drop or an explicit file browser option. Either way, the system sits there and waits until a file is selected.
• Next, we need the file to upload. If it begins immediately - like with Gmail - it probably fires from the previous event. Otherwise, it's likely from a user hitting a "Send" or "Done" button.
• Next, we'd like to provide feedback to the user. Since we know the file's size and we can find out how much has been sent, we can calculate a %-complete to update a progress bar.
• Finally, we need to know when the upload is complete. Most likely we have some sort of "WoohooFileUploadComplete" Event we can capture and let the user know. Hopefully, we'll also get back messages from our destination with error messages/status and we can pass them along for the user.

When you stop to consider it, many of the tasks our code performs can be broken down the same way. I believe the primary reason we don't think of it is because the vast majority of our code is entirely on the backend without the ability to even see or capture our events. Obviously, client side technology like Javascript blurs the line, but the concept is the same. If you're venturing into this world, read up on the Publisher/Subscriber Pattern and the Observer Pattern and figure out how to make your code - both on the frontend and back - play nicely together. Jason Sweat's book on PHP Design Patterns is a good place to start.

* The Live Coding in Flex session went amazingly well. I got a number of great questions and a few people were following along typing exactly what I was as I went. Yes, I had some minor technical issues but the audience was incredibly helpful in pointing out my mistakes. Over. And over. And over again.

** The Publisher/Subscriber model actually applies to most if not all desktop development. It's also familiar to the Javascript guys who work with onClick, onLoad, and all those familiar event-related sounding actions.

Disclosure: Through Blue Parabola, we're working with Adobe and Flex in a number of things. Regardless, I choose to explore Flex for the first time long before that.

I, along with others, have written on interfaces many times before but recently I had occasion to find a new thing about them that makes them really awesome. A few days ago I was tasked with implementing PHPUnit against a Zend Framework application. This application, like many others, makes use of Zend_Auth, and in doing [...]

Both of the lifestreaming projects discussed in this Ars article are both built using PHP frameworks, one Zend and the other is CodeIgniter.  I was pleasantly surprised, I expected them to be Python or Rails projects.

In days of yore, all a self-respecting Internet enthusiast needed to be at the forefront of Web hipness was a simple vanity page with pictures of their dog and maybe a few moderately coherent rants about technology or politics. In the modern world of Web 2.0, where our thoughts and activities are strewn across the global network of tubes, we need slightly more sophisticated solutions to capture and convey the ethos of our Internet identities.

Make your own lifestream with open source Storytlr

If you're looking to build your own style plugins for Drupal's Panels module, the following post will step you through how to do so. The functions and files you'll need to declare are a bit arcane and this beats stepping through the panels code yourself.  I've found the rounded corners plugin that ships with panels is a good reference to review when you're stuck.

Creating panels styles can be very powerful. You can define certain styles for your client to choose from, so they can choose what type of display the panel pane will be like. This way you keep the workflow clean, your code under revision control, your themer gets to keep his sanity, and your concious stays clear.

How to create a panels style plugin | manuee – Putting the puzzle together.

Please ignore this post, which makes the utterly ridiculous recommendation that customers should care about what tools developers use in their work.  Programming is such an individualized task that what works for one person will not necessarily work for someone else.  

That's why, when talking about tools, I like to describe why *I* like the tools and how it makes my tasks easier.  I've tried a lot of different IDEs over the years, and I always come back to jedit.  Some of it is definitely due to comfort, but big IDEs tend to make assumptions about how and where you organize your code, and want you to do all of your work via the IDE.  I like to drop to the command line a lot, to do SVN diffs, commits, and updates.  It feels faster to me, even if it isn't, plus those are the tools I have at my disposal on my server's command line - no fancy GUIs there to guide you.

If the answer is notepad++ you should stay away from that company, no matter how cool the flash design looks.

When evaluating a development shop, you're much better off asking them how they score on the Joel test. It gives you a much more complete look at "the quality of a software team." But Oscar, #9 says "Do you use the best tools money can buy?"  As I read the question, its really asking if you provide them with the best tools to keep them productive and happy.  If someone is a wizard with notepad++ or vim, who's to say they must use anything else?

The recent revelation that at Microsoft, the top developers prefer Old School methods highlights that fancy programming environments aren't necessarily better or make for better programmers.

"When there are five things on the screen, you can burp that out [in text]. But when there are 500 things, [graphical programming] is completely unusable. You zoom in and zoom out and you lose all context. I think it's just smokin' dope."