Website of Jamie McClelland

Avoiding Apache Max Request Workers Errors

Fri, 11 Jul 2025 08:27:10 -0400

[Update 2025-10-09… turns out my first great solution doesn’t work after all, updated to reflect new options.]

Wow, I hate this error:

AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Or, the variation:

AH03490: scoreboard is full, not at MaxRequestWorkers. Increase ServerLimit.

For starters, it means I have to relearn how MaxRequestWorkers functions in Apache:

For threaded and hybrid servers (e.g. event or worker), MaxRequestWorkers restricts the total number of threads that will be available to serve clients. For hybrid MPMs, the default value is 16 (ServerLimit) multiplied by the value of 25 (ThreadsPerChild). Therefore, to increase MaxRequestWorkers to a value that requires more than 16 processes, you must also raise ServerLimit.

Ok… remind me what ServerLimit refers to?

For the prefork MPM, this directive sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. For the worker and event MPMs, this directive in combination with ThreadLimit sets the maximum configured value for MaxRequestWorkers for the lifetime of the Apache httpd process. For the event MPM, this directive also defines how many old server processes may keep running and finish processing open connections. Any attempts to change this directive during a restart will be ignored, but MaxRequestWorkers can be modified during a restart. Special care must be taken when using this directive. If ServerLimit is set to a value much higher than necessary, extra, unused shared memory will be allocated. If both ServerLimit and MaxRequestWorkers are set to values higher than the system can handle, Apache httpd may not start or the system may become unstable. With the prefork MPM, use this directive only if you need to set MaxRequestWorkers higher than 256 (default). Do not set the value of this directive any higher than what you might want to set MaxRequestWorkers to. With worker, use this directive only if your MaxRequestWorkers and ThreadsPerChild settings require more than 16 server processes (default). Do not set the value of this directive any higher than the number of server processes required by what you may want for MaxRequestWorkers and ThreadsPerChild. With event, increase this directive if the process number defined by your MaxRequestWorkers and ThreadsPerChild settings, plus the number of gracefully shutting down processes, is more than 16 server processes (default).

Got it? In other words, you can “consider” raising the MaxRequestWorkers setting all you want, but you can’t just change that setting, you have to read about several other compliated settings, do some math, and spend a lot of time wondering if you are going to remember what you just did and how to undo it if you blow up your server.

On the plus side, typically, nobody should increase this limit - because if the server runs out of connections, it usually means something else is wrong.

In our case, on a shared web server running Apache2 and PHP-FPM, it’s usually because a single web site has gone out of control.

But wait! How can that happen, we are using PHP-FPM’s max_children setting to prevent a single PHP web site from taking down the server?

After years of struggling with this problem I have finally made some headway.

Our PHP pool configuration typically looks like this:

user = site342999writer
group = site342999writer
listen = /run/php/8.1-site342999.sock
listen.owner = www-data
listen.group = www-data
pm = ondemand
pm.max_children = 12
pm.max_requests = 500
php_admin_value[memory_limit] = 256M

And we invoke PHP-FPM via this apache snippet:

<FilesMatch \.php$>
        SetHandler "proxy:unix:/var/run/php/8.1-site342999.sock|fcgi://localhost"
</FilesMatch>

With these settings in place, what happens when we use up all 12 max_children?

According to the docs:

By default, mod_proxy will allow and retain the maximum number of connections that could be used simultaneously by that web server child process. Use the max parameter to reduce the number from the default. The pool of connections is maintained per web server child process, and max and other settings are not coordinated among all child processes, except when only one child process is allowed by configuration or MPM design.

The max parameter seems to default to the ThreadsPerChild, so it seems that the default here is to allow any web site to consume ThreadsPerChild (25) x ServerLimit (16), which is also the max number of over all connections. Not great.

To make matter worse, there is another setting available which is mysteriously called acquire:

If set, this will be the maximum time to wait for a free connection in the connection pool, in milliseconds. If there are no free connections in the pool, the Apache httpd will return SERVER_BUSY status to the client.

By default this is not set which seems to suggest Apache will just hang on to connections forever until a free PHP process becomes available (or some other time out happens).

So, let’s try something different:

 <Proxy "fcgi://localhost">
    ProxySet acquire=1 max=12
  </proxy>

This snippet seems to be the way you can configure the proxy configuration we setup in the SetHandler statement above. It’s documented on the Apache mod_proxy page.

Unfortunately, this does not work. I tried all kinds of different combinations but my best guess is that the use of max and acquire are reserved for tcp connections not unix socket connections, so the only way to achieve this would be to switch our PHP FPM configuration to work over 127.0.0.1 instead of unix sockets, which would bring it’s own problems.

Now what?

I can see two options:

We already pass all traffic via an nginx proxy before it even hits one of our apache back end servers. So, rather than configure just one nginx upstream, we can assign each site their very own upstream with their very own max_conn settings. It feels ugly and wasteful to have one upstream per site on a shared server, but it works.
Install an unsupported apache module. I found mod_vhost_limit, whose very existence seems to confirm my failed struggle at getting this to work. It was written for Redhat and hasn’t been touched in 5 years, but I managed to get it to work on Debian Trixie without much effort. And when I tested, it worked on the first try.

AI's Actual Impact

Fri, 21 Mar 2025 08:27:10 -0400

Two years after OpenAI launched ChatGPT 3.5, humanity is not on the cusp of extinction and Elon Musk seems more responsible for job loss than any AI agent.

However, ask any web administrator and you will learn that large language models are having a significant impact on the world wide web (or, for a less technical account, see Forbes articles on bots). At May First, a membership organization that has been supporting thousands of web site for over 20 years, we have never seen anything like this before.

Wow, my site is getting really popular

It started in 2023. Web sites that performed quite well with a steady viewership started having traffic spikes. These were relatively easy to diagnose, since most of the spikes came from visitors that properly identified themselves as bots, allowing us to see that the big players - OpenAI, Bing, Google, Facebook - were increasing their efforts to scrape as much content from web sites as possible.

Small brochure sites were mostly unaffected because they could be scraped in a matter of minutes. But large sites with an archive of high quality human written content were getting hammered. Any web site with a search feature or a calendar or any interface that generated exponential hits that could be followed were particularly vulnerable.

But hey, that’s what robots.txt is for, right? To tell robots to back off if you don’t want them scraping your site?

Eventually, the cracks began to show. Bots were ignoring robots.txt (did they ever pay that much attention to it in the first place?). Furthermore, rate limiting requests by user agent also began to fail. When you post a link on Facebook, a bot identifying itself as “facebooketernalhit” is invoked to preview the page so it can show a picture and other meta data. We don’t want to rate limit that bot, right? Except, Facebook is also using this bot to scrape your site, often bringing your site to its knees. And don’t get me started on TwitterBot.

Eventually, it became clear that the majority of the armies of bots scraping our sites have completely given up on identifying themselves as bots and are instead using user agents indistinguishable from regular browsers. By using thousands of different IP addresses, it has become really hard to separate the real humans from the bots.

Now what?

So, no, unfortunately, your web site is not suddenly getting really popular. And, you are blessed with a whole new set of strategic decisions.

Fortunately, May First has undergone a major infrastructure transition, resulting in centralized logging of all web sites and a fleet of web proxy servers that intercept all web traffic. Centralized logging means we can analyze traffic and identify bots more easily, and a web proxy fleet allows us to more easily implement rules across all web sites.

However, even with all of our latest changes and hours upon hours of work to keep out the bots, our members are facing some hard decisions about maintaining an open web.

One member of May First provides Google translations of their web site to every language available. But wow, that is now a disaster because instead of having every bot under the sun scrapping all 843 (a made up number) pieces of unique content on their site, the same bots are scraping 843 * (number of available languages) pieces of content on their site. Should they stop providing this translation service in order to ensure people can access their site in the site’s primary language?

Should web sites turn off their search features that include drop down options of categories to prevent bots from systematically refreshing the search page with every possible combination of search terms?

Do we need to alter our calendar software to avoid providing endless links into the future (ok, that is an easy one)?

What’s next?

Something has to change.

Lock down web 2.0. Web 2.0 brought us wonderful dynamic web sites, which Drupal and WordPress and many other pieces of amazing software have supported for over a decade. This is the software that is getting bogged down by bots. Maybe we need to figure out a way to lock down the dynamic aspects of this software to logged in users and provide static content for everyone else?
Paywalls and accounts everywhere. There’s always been an amazing non-financial reward to providing a web site with high quality movement oriented content for free. It populates the search engines, provides links to inspiring and useful content in moments of crises, and can galvanize movements. But these moments of triumph happen between long periods of hard labor that now seems to mostly feed capitalist AI scumbags. If we add a new set of expenses and labor to keep the sites running for this purpose, how sustainable is that? Will our treasure of free movement content have to move behind paywalls or logins? If we provide logins, will that keep the bots out or just create a small hurdle for them to automate the account creation process? What happens when we can’t search for this kind of content via search engines?
Cutting deals. What if our movement content providers are forced to cut deals with the AI entrepreneurs to allow the paying scumbags to fund the content creation. Eww. Enough said.
Bot detection. Maybe we just need to get better at bot detection? This will surely be an arms race, but would have some good benefits. Bots have also been filling out our forms and populating our databases with spam, testing credit cards against our donation pages, conducting denial of service attacks and all kinds of other irritating acts of vandalism. If we were better at stopping bots automatically it would have a lot of benefits. But what impact would it have on our web sites and the experience of using them? What about “good” bots (RSS feed readers, payment processors, web hooks, uptime detectors)? Will we cut the legs off any developer trying to automate something?

I’m not really sure where this is going, but it seems that the world wide web is about to head in a new direction.

How do I warm up an IP Address?

Sat, 21 Sep 2024 08:27:10 -0400

After years on the waiting list, May First was just given a /24 block of IP addresses. Excellent.

Now we want to start using them for, among other things, sending email.

I haven’t added a new IP address to our mail relays in a while and things seems to change regularly in the world of email so I’m curious: what’s the best 2024 way to warm up IP addresses, particularly using postfix?

Sendergrid has a nice page on the topic. It establishes the number of messages to send per day. But I’m not entirely sure how to fit messages per day into our setup.

We use round robin DNS to direct email to one of several dozen email relay servers using postfix. And unfortunately our DNS software (knot) doesn’t have a way to add weights to ensure some IPs show up more often than others (much less limit the specific number of messages a given relay should get).

Postfix has some nice knobs for rate limiting, particularly: default_destination_recipient_limit and default_destination_rate_delay

If default_destination_recipient_limit is over 1, then default_destination_rate_delay is equal to the minimum delay between sending email to the same domain.

So, I’m staring our IP addresses out at 30m - which prevents any single domain from receiving more than 2 messages per hour. Sadly, there are a lot of different domain names that deliver to the same set of popular corporate MX servers, so I am not sure I can accurately control how many messages a given provider sees coming from a given IP address. But it’s a start.

A bigger problem is that messages that exceed the limit hang out in the active queue until they can be sent without violating the rate limit. Since I can’t fully control the number of messages a given queue receives (due to my inability to control the DNS round robin weights), a lot of messages are going to be severely delayed, especially ones with an @gmail.com domain name.

I know I can temporarily set relayhost to a different queue and flush deferred messages, however, as far as I can tell, it doesn’t work with active messages.

To help mitigate the problem I’m only using our bulk mail queue to warm up IPs, but really, this is not ideal.

Suggestions welcome!

Update #1

If you are running postfix in a multi-instance setup and you have instances that are already warmed up, you can move active messages between queues with these steps:

# Put the message on hold in the warming up instance
postsuper -c /etc/postfix-warmingup -h $queueid
# Copy to a warmed up instance
cp --preserve=mode,ownership,timestamp /var/spool/postfix-warmingup/hold/$queueid /var/spool/postfix-warmedup/incoming/
# Queue the message
postqueue -c /etc/postfix-warmedup -i $queueid
# Delete from the original queue.
postsuper -c /etc/postfix-warmingup -d $queueid

After just 12 hours we had thousands of messages piling up. This warm up method was never going to work without the ability to move them to a faster queue.

[Additional update: be sure to reload the postfix instance after flushing the queue so messages are drained from the active queue on the correct schedule. See update #4.]

Update #2

After 24 hours, most email is being accepted as far as I can tell. I am still getting a small percentage of email deferred by Yahoo with:

421 4.7.0 [TSS04] Messages from 204.19.241.9 temporarily deferred due to unexpected volume or user complaints - 4.16.55.1; see https://postmaster.yahooinc.com/error-codes (in reply

So I will keep it as 30m for another 24 hours or so and then move to 15m. Now that I can flush the backlog of active messages I am in less of a hurry.

Update #3

Well, this doesn’t seem to be working the way I want it to.

When a message arrives faster than the designated rate limit, it remains in the active queue.

I’m entirely sure how the timing is supposed to work, but at this point I’m down to a 5m rate delay, and the active messages are just hanging out for a lot longer than 5m. I tried flushing the queue, but that only seems to affect the deferred messages. I finally got them re-tried with systemctl reload. I wonder if there is a setting to control this retry? Or better yet, why can’t these messages that exceed the rate delayed be deferred instead?

Update #4

I think I see why I was confused in Update #3 about the timing. I suspect that when I move messages out of the active queue it screws up the timer. Reloading the instance resets the timer. Every time you muck with active messages, you should reload.

Gmail vs Tor vs Privacy

Wed, 18 Sep 2024 08:27:10 -0400

A legit email went to spam. Here are the redacted, relevant headers:

[redacted]
X-Spam-Flag: YES
X-Spam-Level: ******
X-Spam-Status: Yes, score=6.3 required=5.0 tests=DKIM_SIGNED,DKIM_VALID,
[redacted]
	*  1.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL
	*      [185.220.101.64 listed in xxxxxxxxxxxxx.zen.dq.spamhaus.net]
	*  3.0 RCVD_IN_SBL_CSS Received via a relay in Spamhaus SBL-CSS
	*  2.5 RCVD_IN_AUTHBL Received via a relay in Spamhaus AuthBL
	*  0.0 RCVD_IN_PBL Received via a relay in Spamhaus PBL
[redacted]
[very first received line follows...]
Received: from [10.137.0.13] ([185.220.101.64])
        by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-378956d2ee6sm12487760f8f.83.2024.09.11.15.05.52
        for <xxxxx@mayfirst.org>
        (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
        Wed, 11 Sep 2024 15:05:53 -0700 (PDT)

At first I though a Gmail IP address was listed in spamhaus - I even opened a ticket. But then I realized it wasn’t the last hop that Spamaus is complaining about, it’s the first hop, specifically the ip 185.220.101.64 which appears to be a Tor exit node.

The sender is using their own client to relay email directly to Gmail. Like any sane person, they don’t trust Gmail to protect their privacy, so they are sending via Tor. But WTF, Gmail is not stripping the sending IP address from the header.

I’m a big fan of harm reduction and have always considered using your own client to relay email with Gmail as a nice way to avoid some of the surveillance tax Google imposes.

However, it seems that if you pursue this option you have two unpleasant choices:

Embed your IP address in every email message or
Use Tor and have your email messages go to spam

I supposed you could also use a VPN, but I doubt the IP reputation of most VPN exit nodes are going to be more reliable than Tor.

MariaDB mystery

Wed, 11 Sep 2024 08:27:10 -0400

I keep getting an error in our backup logs:

Sep 11 05:08:03 Warning: mysqldump: Error 2013: Lost connection to server during query when dumping table `1C4Uonkwhe_options` at row: 1402
Sep 11 05:08:03 Warning: Failed to dump mysql databases ic_wp

It’s a WordPress database having trouble dumping the options table.

The error log has a corresponding message:

Sep 11 13:50:11 mysql007 mariadbd[580]: 2024-09-11 13:50:11 69577 [Warning] Aborted connection 69577 to db: 'ic_wp' user: 'root' host: 'localhost' (Got an error writing communication packets)

The Internet is full of suggestions, almost all of which either focus on the network connection between the client and the server or the FEDERATED plugin. We aren’t using the federated plugin and this error happens when conneting via the socket.

Check it out - what is better than a consistently reproducible problem!

It happens if I try to select all the values in the table:

root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options' ic_wp > /dev/null
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#

It happens when I specifiy one specific offset:

root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#

It happens if I specify the field name explicitly:

root@mysql007:~# mysql --protocol=socket -e 'select option_id,option_name,option_value,autoload from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#

It doesn’t happen if I specify the key field:

root@mysql007:~# mysql --protocol=socket -e 'select option_id from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
+-----------+
| option_id |
+-----------+
|  16296351 |
+-----------+
root@mysql007:~#

It does happen if I specify the value field:

root@mysql007:~# mysql --protocol=socket -e 'select option_value from 1C4Uonkwhe_options limit 1 offset 1402' ic_wp
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#

It doesn’t happen if I query the specific row by key field:

root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
+-----------+----------------------+--------------+----------+
| option_id | option_name          | option_value | autoload |
+-----------+----------------------+--------------+----------+
|  16296351 | z_taxonomy_image8905 |              | yes      |
+-----------+----------------------+--------------+----------+
root@mysql007:~#

Hm. Surely there is some funky non-printing character in that option_value right?

root@mysql007:~# mysql --protocol=socket -e 'select CHAR_LENGTH(option_value) from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
+---------------------------+
| CHAR_LENGTH(option_value) |
+---------------------------+
|                         0 |
+---------------------------+
root@mysql007:~# mysql --protocol=socket -e 'select HEX(option_value) from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
+-------------------+
| HEX(option_value) |
+-------------------+
|                   |
+-------------------+
root@mysql007:~#

Resetting the value to an empty value doesn’t make a difference:

root@mysql007:~# mysql --protocol=socket -e 'update 1C4Uonkwhe_options set option_value = "" where option_id = 16296351' ic_wp
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options' ic_wp > /dev/null
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#

Deleting the row in question causes the error to specify a new offset:

root@mysql007:~# mysql --protocol=socket -e 'delete from 1C4Uonkwhe_options where option_id = 16296351' ic_wp
root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options' ic_wp > /dev/null
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~# mysqldump ic_wp > /dev/null
mysqldump: Error 2013: Lost connection to server during query when dumping table `1C4Uonkwhe_options` at row: 1401
root@mysql007:~#

If I put the record I deleted back in, we return to the old offset:

root@mysql007:~# mysql --protocol=socket -e 'insert into 1C4Uonkwhe_options VALUES(16296351,"z_taxonomy_image8905","","yes");' ic_wp 
root@mysql007:~# mysqldump ic_wp > /dev/null
mysqldump: Error 2013: Lost connection to server during query when dumping table `1C4Uonkwhe_options` at row: 1402
root@mysql007:~#

I’m losing my little mind. Let’s get drastic and create a whole new table, copy over the data delicately working around the deadly offset:

oot@mysql007:~# mysql --protocol=socket -e 'create table 1C4Uonkwhe_new_options like 1C4Uonkwhe_options;' ic_wp 
root@mysql007:~# mysql --protocol=socket -e 'insert into 1C4Uonkwhe_new_options select * from 1C4Uonkwhe_options limit 1402 offset 0;' ic_wp 
--- There is only 33 more records, not sure how to specify unlimited limit but 100 does the trick.
root@mysql007:~# mysql --protocol=socket -e 'insert into 1C4Uonkwhe_new_options select * from 1C4Uonkwhe_options limit 100 offset 1403;' ic_wp

Now let’s make sure all is working properly:

root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_new_options' ic_wp >/dev/null;

Now let’s examine which row we are missing:

root@mysql007:~# mysql --protocol=socket -e 'select option_id from 1C4Uonkwhe_options where option_id not in (select option_id from 1C4Uonkwhe_new_options) ;' ic_wp 
+-----------+
| option_id |
+-----------+
|  18405297 |
+-----------+
root@mysql007:~#

Wait, what? I was expecting option_id 16296351.

Oh, now we are getting somewhere. And I see my mistake: when using offsets, you need to use ORDER BY or you won’t get consistent results.

root@mysql007:~# mysql --protocol=socket -e 'select option_id from 1C4Uonkwhe_options order by option_id limit 1 offset 1402' ic_wp ;
+-----------+
| option_id |
+-----------+
|  18405297 |
+-----------+
root@mysql007:~#

Now that I have the correct row… what is in it:

root@mysql007:~# mysql --protocol=socket -e 'select * from 1C4Uonkwhe_options where option_id = 18405297' ic_wp ;
ERROR 2013 (HY000) at line 1: Lost connection to server during query
root@mysql007:~#

Well, that makes a lot more sense. Let’s start over with examining the value:

root@mysql007:~# mysql --protocol=socket -e 'select CHAR_LENGTH(option_value) from 1C4Uonkwhe_options where option_id = 18405297' ic_wp ;
+---------------------------+
| CHAR_LENGTH(option_value) |
+---------------------------+
|                  50814767 |
+---------------------------+
root@mysql007:~#

Wow, that’s a lot of characters. If it were a book, it would be 35,000 pages long (I just discovered this site). It’s a LONGTEXT field so it should be able to handle it. But now I have a better idea of what could be going wrong. The name of the option is “rewrite_rules” so it seems like something is going wrong with the generation of that option.

I imagine there is some tweak I can make to allow MariaDB to cough up the value (read_buffer_size? tmp_table_size?). But I’ll start with checking in with the database owner because I don’t think 35,000 pages of rewrite rules is appropriate for any site.

Who ate my RAM?

Wed, 07 Aug 2024 08:27:10 -0400

One of our newest servers, with a hefty 256GB of RAM, recently began killing processes via the oomkiller.

According to free, only half of the RAM was in use (125GB). About 4GB was free, with the remainer used by the file cache.

I’m used to seeing unexpected “free RAM” numbers like this and have been assured that the kernel is simply not wasting RAM. If it’s not needed, use it to cache files to save on disk I/O. That make sense.

However… why is the oomkiller being called instead of flushing the file cache?

I came up with all kinds of amazing and wrong theories: maybe the RAM is fragmented (is that even a thing?!?), maybe there is a spike in RAM and the kernel can’t flush the cache quickly enough (I really don’t think that’s a thing). Maybe our kvm-manager has a weird bug (nope, but that didn’t stop me from opening a spurious bug report).

I learned lots of cool things, like the oomkiller report includes a table of the memory in use by each process (via the rss column) - and you have to muliply that number by 4096 because it’s in 4K pages.

That’s how I discovered that the oomkiller was killing off processes with only half the memory in use.

I also learned that lsof sometimes lists the same open file multiple times, which made me think a bunch of files were being opened repeatedly causing a memory problem, but really it amounted to nothing.

That last thing I learned, courtesy of an askubuntu post is that the /dev filesystem is allocated by default exactly half the RAM on the system. What a coincidence! That is exactly how much RAM is useable on the server.

And, on the server in question, that filesystem is full. What?!? Normally, that filesystem should be using 0 bytes because it’s not a real filesystem. But in our case a process created a 127GB file there - it was only stopped because the file system filled up.

Users without passwords

Sun, 22 Oct 2023 08:27:10 -0400

About fifteen years ago, while debugging a database probem, I was horrified to discover that we had two root users - one with the password I had been using and one without a password. Nooo!

So, I wrote a simple maintenance script that searched for and deleted any user in our database without a password. I even made it part of our puppet recipe - since the database server was in use by users and I didn’t want anyone using SQL statements to change their password to an empty value.

Then I forgot about it.

Recently, I upgraded our MariaDB databases to Debian bullseye, which inserted the mariadb.sys user which…. doesn’t have a password set. It seems to be locked down in other ways, but my dumb script didn’t know about that and happily deleted the user.

Who needs that mariadb.sys user anyway?

Apparently we all do. On one server, I can’t login as root anymore. On another server I can login as root, but if I try to list users I get an error:

ERROR 1449 (HY000): The user specified as a definer (‘mariadb.sys’@’localhost’) does not exist

The Internt is full of useless advice. The most common is to simply insert that user. Except…

MariaDB [mysql]> CREATE USER `mariadb.sys`@`localhost` ACCOUNT LOCK PASSWORD EXPIRE;
ERROR 1396 (HY000): Operation CREATE USER failed for 'mariadb.sys'@'localhost'
MariaDB [mysql]>

Yeah, that’s not going to work.

It seems like we are dealing with two changes. One, the old mysql.user table was replaced by the global_priv table and then turned into a view for backwards compatibility.

And two, for sensible reasons the default definer for this view has been changed from the root user to a user that, ahem, is unlikely to be changed or deleted.

Apparently I can’t add the mariadb.sys user because it would alter the user view which has a definer that doesn’t exist. Although not sure if this really is the reason?

Fortunately, I found an excellent suggestion for changing the definer of a view. My modified version of the answer is, run the following command which will generate a SQL statement:

SELECT CONCAT("ALTER DEFINER=root@localhost VIEW ", table_name, " AS ", view_definition, ";") FROM information_schema.views WHERE table_schema='mysql' AND definer = 'mariadb.sys@localhost';

Then, execute the statement.

And then also update the mysql.proc table:

UPDATE mysql.proc SET definer = 'root@localhost' WHERE definer = 'mariadb.sys@localhost';

And lastly, I had to run:

DELETE FROM tables_priv WHERE User = 'mariadb.sys';
FLUSH privileges;

Wait, was the tables_priv entry the whole problem all along? Not sure. But now I can run:

CREATE USER `mariadb.sys`@`localhost` ACCOUNT LOCK PASSWORD EXPIRE;
GRANT SELECT, DELETE ON `mysql`.`global_priv` TO `mariadb.sys`@`localhost`;

And reverse the other statements:

SELECT CONCAT("ALTER DEFINER=`mariadb.sys`@localhost VIEW ", table_name, " AS ", view_definition, ";") FROM information_schema.views WHERE table_schema='mysql' AND definer = 'root@localhost';

[Execute the output.]

UPDATE mysql.proc SET definer = 'mariadb.sys@localhost' WHERE definer = 'root@localhost';

And while we’re on the topic of borked MariaDB authentication, here are the steps to change the root password and restore all root privielges if you can’t get in at all or your root user is missing the GRANT OPTION (you can change “ALTER” to “CREATE” if the root user does not even exist):

systemctl stop mariadb
mariadbd-safe --skip-grant-tables --skip-networking &
mysql -u root
[mysql]> FLUSH PRIVILEGES
[mysql]> ALTER USER `root`@`localhost` IDENTIFIED VIA mysql_native_password USING PASSWORD('your-secret-password') OR unix_socket; 
[mysql]> GRANT ALL PRIVILEGES ON *.* to 'root'@'localhost' WITH GRANT OPTION;
mariadbd-admin shutdown
systemctl start mariadb

What am I missing about AI?

Tue, 18 Jul 2023 08:27:10 -0400

Last month I blogged about how the mainstream media is focusing on the wrong parts of the Artificial Intelligence/ChatGPT story.

One of the comments left on the post was:

I encourage you to dig a little deeper. If LLM’s were just probability
machines, no one would be raising any flags.

Hinton, Bengio, Tegmark and many others are not simpletons. It is the fact that
the architecture and specific training (deep NN, back prop / gradient descend)
produces a system with emergent properties, beyond just a probability machine,
when the system size reaches some thresholds, that has them spooked.

They do understand mathematics and stats and probabilities, i assure you. It is
just that you may have only read the layman’s articles and not the scientific
ones

I confess: I haven’t made much progress in this regard. I gave Vicky Boykis' Embeddings a go, and started to get a handle on the math, but honestly had a hard time following it. I’m open to suggestions from anyone with a few good recommendations for scientific papers accessible to non-math professionals, particularly ones that explain the “emergent” properties and what that means.

Meanwhile, regardless of the scientific truths or falsehoods around chat GPT, the mainstream media continues to miserably fail in helping the rest of us understand the implications of this technology.

Most recently, I listend to This American Life’s “First Contact” (part of their “Greetings People of Earth” show).

They interviewed several Microsft AI researchers who first experimented with ChatGPT 4 prior to it’s big release.

The focus of the researchers was: can we demonstrate chat GPT’s general intelligence ability by presenting it with logic problems it could not possibly have encountered before? And the answer: YES!

The two examples were:

Stacking: the researcher asked chat GPT how to stack a number of odd objects in a stable way (a book, a dozen eggs, a nail, etc) and chat GPT gave both the correct answer and a reasonable explanation of why.
Hidden state: the researcher described two people in a room with a cat. One person put the cat in a basket and left. The other moved the cat to a box ad left. And, remarkably, chat GPT could explain that when they returned, the first person would think the cat is in the basket and the second person would know it’s in the box.

I thought this was pretty cool. So I fired up chat GPT (and even ponied up for chat GPT version 4). I asked it my own stacking question and, hm, chat GPT thought a plate should be placed on top of a can of soda instead of beneath it. So, well, mostly right but I’m pretty sure any reasonable human would put the can of soda on the plate not the other way around (chat GPT 3.5 wanted the can of soda to be balanced on the tip of the nail).

I then asked it my own simple version of the cat problem and it got it right. Very good. But when I asked it a much more complicated and weird version of the cat problem (involving beetles in a mansion with a movie theater and changing movies and a butler with a big mustache) it got the answer flat out wrong.

Did anyone at This American Life try this? Really? It seems like a basic responsibility of journalism to fact check the experts. Maybe the scientists would have had a convincing response? Or maybe scientists are just like everyone else and can get caught up in the excitement and make mistakes?

I am amazed and awed by what chat GPT can do - it truly is remarkable. And, I think that a lot of human intelligence is synthesizing what we’ve seen and simply regurgitating it in a different context - a task that chat GPT is way better at doing than we are.

But the overriding message of most mainstream media stories is that chat GPT is somehow going beyond word synthesis and probability and magically tapping into a form of logic. If the scientific papers are demonstrating this remarkable feat, I think the media needs to do a way better job reporting it.

Enough about the AI Apocalypse Already

Thu, 01 Jun 2023 08:27:10 -0400

After watching Democracy Now’s segment on artificial intelligence I started to wonder - am I out of step on this topic?

When people claim artificial intelligence will surpass human intelligence and thus threaten humanity with extinction, they seem to be referring specifically to advances made with large language models.

As I understand them, large language models are probability machines that have ingested massive amounts of text scraped from the Internet. They answer questions based on the probability of one series of words (their answer) following another series of words (the question).

It seems like a stretch to call this intelligence, but if we accept that definition then it follows that this kind of intelligence is nothing remotely like human intelligence, which makes the claim that it will surpass human intelligence confusing. Hasn’t this kind of machine learning surpassed us decades ago?

Or when we say “surpass” does that simply refer to fooling people into thinking an AI machine is a human via conversation? That is an important milestone, but I’m not ready to accept the turing test as proof of equal intelligence.

Furthermore, large language models “hallucinate” and also reflect the biases of their training data. The word “hallucinate” seems like a euphemism, as if it could be corrected with the right medication when in fact it seems hard to avoid when your strategy is to correlate words based on probability. But even if you could solve the “here is a completely wrong answer presented with sociopathic confidence” problem, reflecting the biases of your data sources seems fairly intractable. In what world would a system with built-in bias be considered on the brink of surpassing human intelligence?

The danger from LLMs seems to be their ability to convince people that their answers are correct, including their patently wrong and/or biased answers.

Why do people think they are giving correct answers? Oh right… terrifying right wing billionaires (with terrifying agendas have been claiming AI will exceed human intelligence and threaten humanity and every time they sign a hyperbolic statement they get front page mainstream coverage. And even progressive news outlets are spreading this narrative with minimal space for contrary opinions (thank you Tawana Petty from the Algorithmic Justice League for providing the only glimpse of reason in the segment).

The belief that artificial intelligence is or will soon become omnipotent has real world harms today: specifically it creates the misperception that current LLMs are accurate, which paves the way for greater adoption among police forces, social service agencies, medical facilities and other places where racial and economic biases have life and death consequences.

When the CEO of OpenAI calls the technology dangerous and in need of regulation, he gets both free advertising promoting the power and supposed accuracy of his product and the possibility of freezing further developments in the field that might challenge OpenAI’s current dominance.

The real threat to humanity is not AI, it’s massive inequality and the use of tactics ranging from mundane bureaucracy to deadly force and incarceration to segregate the affluent from the growing number of people unable to make ends meet. We have spent decades training bureaucrats, judges and cops to robotically follow biased laws to maintain this order without compassion or empathy. Replacing them with AI would be make things worse and should be stopped. But, let’s be clear, the narrative that AI is poised to surpass human intelligence and make humanity extinct is a dangerous distraction that runs counter to a much more important story about “the very real and very present exploitative practices of the [companies building AI], who are rapidly centralizing power and increasing social inequities.”.

Maybe we should talk about that instead?

Cranky old timers should know perl

Wed, 17 May 2023 08:27:10 -0400

I act like an old timer (I’ve been around linux for 25 years and I’m cranky about new tech that is not easily maintained and upgraded) yet somehow I don’t know perl. How did that happen?

I discovered this state when I decided to move from the heroically packaged yet seemingly upstream un-maintained opendmarc package to authentication_milter.

It’s written in perl. And, alas, not in debian.

How hard could this be?

The instructions for installing seemed pretty straight forward: cpanm Mail::Milter::Authentication.

Wah. I’m glad I tried this out on a test virtual machine. It took forever! It ran tests! It compiled things! And, it installed a bunch of perl modules already packaged in Debian.

I don’t think I want to add this command to my ansible playbook.

Next I spent an inordinate amount of time trying to figure out how to list the dependencies of a given CPAN module. I was looking for something like cpanm --list-dependencies Mail::Milter::Authentication but eventually ended up writing a perl script that output perl code, inserting a “use " before each dependency and a semicolon and line break after them. Then, I could execute that script on a clean debian installation and see which perl modules I needed. For each error, I checked for the module in Debian (and installed it) or kept a list of modules I would have to build (and commented out the line).

Once I had a list of modules to build, I used the handy cpan2deb command. It took some creative ordering but eventually I got it right. Since I will surely forget how to do this when it’s time to upgrade, I wrote a script.

In total it took me several days to figure this all out, so I once again find myself very appreciative of all the debian packagers out there - particularly the perl ones!!

And also… if I did this all wrong and there is an easier way I would love to hear about it in the comments.

Electron doesn't like negative layout coordinates

Thu, 20 Apr 2023 08:27:10 -0400

I got a second external monitor. Overkill? Probably, but I like having a dedicated space to instant messaging (now left monitor) and also a dedicated space for a web browser (right monitor).

But, when I moved signal-desktop to the left monitor, clicks stopped working. I moved it back to my laptop screen, clicks started working. Other apps (like gajim) worked fine. A real mystery.

I spent a lot of time on the wrong thing. I turned this monitor into portrait mode. Maybe signal doesn’t like portrait mode? Nope.

Maybe signal doesn’t think it has enough horizontal space? Nope.

Maybe signal suddently doesn’t like being on an external monitor? Nope.

Maybe signal will output something useful if I start it via the terminal? Nope (but that was a real distraction).

Then, I discovered that mattermost desktop behaves the same way. A clue! So, now I know it’s an electron app limitation, not a signal limitation.

Finally I hit on the problem: Via my sway desktop, I set my laptop screen to the x,y coordinates: 0,0 (since it’s in the middle). I set the left monitor to negative coordinates so it would appear on the left. Well, electron does not like that, not sure why.

Now, my left monitor occupies 0,0 and the rest are adjusted to that center.

I originally used negative coordinates so when I unplugged my monitors, my laptop would still be 0,0 and display all desktops properly. Fortunately, sway magically figures that all out even when the “center” shifts by unplugging the monitors. Hooray for sway.

Doing whatever Gmail says

Wed, 12 Apr 2023 08:27:10 -0400

As we slowly move our members to our new email infrastructure, an unexpected twist turned up: One member reported getting the Gmail warning:

Be careful with this message The sender hasn’t authenticated this message so Gmail can’t verify that it actually came from them.

They have their email delivered to May First, but have configured Gmail to pull in that email using the “Check mail from other accounts” feature. It worked fine on our old infrastructure, but started giving this message when we transitioned.

A further twist: he only receives this message from email sent by other people in his organization - in other words email sent via May First gets flagged, email sent from other people does not.

These Gmail messages typically warn users about email that has failed (or lacks) both SPF and DKIM. However, before diving into the technical details, my first thought was: why is Gmail giving a warning on a message that wasn’t even delivered to them? It’s always nice to get confirmation from others that this is totally wrong behavior. Unfortunately, when it’s Gmail, it doesn’t matter if they are wrong. We all have to deal with it.

So next, I decided to investigate why this message failed both DKIM (digital signature) and SPF (ensuring the message was sent from an authorized server).

Examining the headers immediately turned up the SPF failure:

Authentication-Results: mx.google.com;
       spf=fail (google.com: domain of xxx@xxx.org does not designate n.n.n.n as permitted sender) smtp.mailfrom=xxx@xxx.org

The IP address Google checked to ensure the message was sent by an authorized server is the IP address of our internal mail filter server in our new email infrastructure. That’s the last hop before delivery to the user’s mailbox, so that’s the last hop Gmail sees. This is why Gmail is totally wrong to run this check: all email messages retreived via their mail fecthing service are going to fail the SPF test because Gmail has no way of knowing what the actual last hop is.

So why is this problem only showing up after we transitioned to our new infrastructure? Because our old infrastructure had only one mail server for every user. The one mail server was the MX server and the relay server, so it was included in their SPF record.

And why does this only affect mail sent via May First and not other domains?

Because we add our DKIM signature to outgoing email, not to email delivered internally. Therefore, these messages both fail the SPF check and also don’t have a DKIM signature. Other messages have a DKIM signature.

Ugggg. So what do we do now? Clearly, something dumb and simple is in order: I added the IP addresses of our internal filter servers to our global SPF record.

Someday, years from now, after Gmail is long gone (or has fixed this dumb behavior), when I’m doing whatever retired people like me do, someone will notice that our internal filter server IPs are included in our SPF record. Hopefully they will fix the problem, but instead they’ll probably think: no idea why these are here - something will probably break if I remove them.

Web caching is hard

Sun, 14 Aug 2022 08:27:10 -0400

Web caching is hard. And also, maybe I’m not that good under pressure? In any event, I made the following mistakes while trying to debug a web site using our nginx cache that bit the dust under heavy load today:

Action: I ran curl -I https://website.org/ and it hung.

Wrong assumption: Something is wrong with nginx. Why else would it just hang?

Reconsidered conclusion: The resource (the home page) is a MISS, so nginx has to retrieve it from the origin, but the origin is over-loaded and timing out, so my request is also timing out. Maybe something is wrong with the nginx caching configuration since the home page really should be a HIT… but that’s another problem.

Action: I changed the configuration from our normal caching set of directives to our aggressive caching set of directives, reloaded nginx and curl -I https://website.org/ still hung.

Wrong assumption: aggressive caching isn’t working and I need a different configuration.

Reconsidered conclusion: The home page still has failed to be loaded from the origin, so every request for it is going to be a MISS, and is going to hang, until nginx is able to fill the cache with it. The configuration change might be the right change; we just need the origin to calm down before we will know.

Action: I restarted PHP on the origin to free up PHP processes so my home page request can fill the cache … and still curl -I https://website.org/ hangs.

Wrong assumption: WTF! The world is ending!

Reconsidered conclusion: The regular traffic which is accessing other pages (not the home page) consumed all the available PHP processes on the origin before my request for the home page could complete, so nginx is still unable to fill the cache with the home page.

Action: Once we got things under control, I changed the caching level from aggressive back down to normal. I ran curl -I https://website.org/ and it was HIT’ing. I concluded that we don’t need the aggressive cache after all. Got some coffee, came back later and ran it again and it consistently showed MISS.

Wrong assumption: What?!? Did something change on the origin to stop the cache from working??

Reconsidered conclusion: The aggressive cache set the cache for 5 minutes. Even after changing to normal caching, the home page was still cached so it was served from the cache. After 5 minutes, the cache expired. Now, the normal cache setting are in play to determine whether the request would be cached or not. In other words, you have to wait for the cache to expire (or bust the cache) before you can effectively know if the new cache settings are working.

Fine tuning Thunderbird's end-to-end encryption

Thu, 04 Aug 2022 18:27:10 -0400

I love that Thunderbird really tackled OpenPGP head on and incorporated it directly into the client. I know it’s been a bit rough for some users, but I think it’s a good long term investment.

And to demonstrate I’ll now complain about a minor issue :).

I replied to an encrypted message but couldn’t send the response using encryption. I got an error message indicating that “End-to-end encryption requires resolving certificate issues for” and it listed the recipient email address.

I spent an enormous amount of time examining the recipient’s OpenPGP key. I made sure it was not expired. I made sure it was actually in my Thunderbird key store not just in my OpenPGP keychain. I made sure I had indicated that I trust it enough to use. I re-downloaded it.

I eventually gave up and didn’t send the email. Then I responded to another encrypted email and it worked. What!?!?

I spent more time comparing the recipients before I realized the problem was the sending address, not the recipient address.

I have an OpenPGP key that lists several identities. I have a Thunderbird Account that uses the Identities feature to add several from addresses. And, it turns out that in Thunderbird, you need to indicate which OpenPGP key to use for your main account… but also for each identity. When you drill down to Manage Identities for your account, you are able to indicate which OpenPGP key you want to use for each identity. Once I indicated that each identity should use my OpenPGP key, the issue was resolved.

And here’s my Thunderbird bug asking for an error message pointing to the sender address, not the recipient address.

Deleting an app won't bring back Roe v Wade

Sat, 25 Jun 2022 18:27:10 -0400

In some ways it feels like 2016 all over again.

I’m seeing panic-stricken calls for everyone to delete their period apps, close their Facebook accounts, de-Google their cell phones and, generally speaking, turn their entire online lives upside down to avoid the techno-surveillance dragnet unleashed by the overturning of Roe v. Wade.

I’m sympathetic and generally agree that many of us should do most of those things on any given day. But, there is a serious problem with this cycle of repression and panic: it’s very bad for organizing.

In our rush to give people concrete steps they can take to feel safer, we’re fueling a frenzy of panic and fear, which seriously inhibits activism.

Now is the time to remind people that, over the last 20 years, a growing movement of organizers and technologists have been building user-driven, privacy-respecting, consentful technology platforms as well as organizations and communities to develop them.

We have an entire eco system of:

technology platform cooperatives,
movement aligned Internet providers that pre-date the founding of Twitter and are still going strong (May First, Riseup, and Autistici just to name a few),
the fediverse, a well developed, de-centralized alterntiave to corporate social media (try Social.coop if you want to get started),
powerful open source, privacy respecting software geared for organizing and movement providers hosting it (see CiviCRM and Progressive Technology Project),
multi-year campaigns targeting poor tech practices of corporate technology giants (see Mijente’s No Tech for ICE),
so many more examples, far too numerous to name.

All of these projects need our love and support over the long haul. Please help spread the word - rather then just deleting an app, let’s encourage people to join an organziation or try out a new kind of technology that will serve us down the road when we may need it even more then today.

A very liberal spam assassin rule

Mon, 20 Jun 2022 08:27:10 -0400

I just sent myself a test message via Powerbase (a hosted CiviCRM project for community organizers) and it didn’t arrive. Wait, nope, there it is in my junk folder with a spam score of 6!

X-Spam-Status: Yes, score=6.093 tagged_above=-999 required=5
	tests=[BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1,
	DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, DMARC_MISSING=0.1,
	HTML_MESSAGE=0.001, KAM_WEBINAR=3.5, KAM_WEBINAR2=3.5,
	NO_DNS_FOR_FROM=0.001, SPF_HELO_NONE=0.001, ST_KGM_DEALS_SUB_11=1.1,
	T_SCC_BODY_TEXT_LINE=-0.01] autolearn=no autolearn_force=no

What just happened?

A careful look at the scores suggest that the KAM_WEBINAR and KAM_WEBINAR2 rules killed me. I’ve never heard of them (this email came through a system I’m not administering). So, I did some searching and found a page with the rules:

# SEMINARS AND WORKSHOPS SPAM
header   __KAM_WEBINAR1 From =~ /education|career|manage|learning|webinar|project|efolder/i
header   __KAM_WEBINAR2 Subject =~ /last chance|increase productivity|workplace morale|payroll dept|trauma.training|case.study|issues|follow.up|service.desk|vip.(lunch|breakfast)|manage.your|private.business|professional.checklist|customers.safer|great.timesaver|prep.course|crash.course|hunger.to.learn|(keys|tips).(to|for).smarter/i
header   __KAM_WEBINAR3 Subject =~ /webinar|strateg|seminar|owners.meeting|webcast|our.\d.new|sales.video/i
body     __KAM_WEBINAR4 /executive.education|contactid|register now|\d+.minute webinar|management.position|supervising.skills|discover.tips|register.early|take.control|marketing.capabilit|drive.more.sales|leveraging.cloud|solution.provider|have.a.handle|plan.to.divest|being.informed|upcoming.webinar|spearfishing.email|increase.revenue|industry.podcast|\d+.in.depth.tips|early.bird.offer|pmp.certified|lunch.briefing/i

meta     KAM_WEBINAR (__KAM_WEBINAR1 + __KAM_WEBINAR2 + __KAM_WEBINAR3 + __KAM_WEBINAR4 >= 3)
describe KAM_WEBINAR Spam for webinars
score    KAM_WEBINAR 3.5

meta     KAM_WEBINAR2 (__KAM_WEBINAR1 + __KAM_WEBINAR2 + __KAM_WEBINAR3 + __KAM_WEBINAR4 >= 4)
describe KAM_WEBINAR2 Spam for webinars
score    KAM_WEBINAR2 3.5

For those of you who don’t care to parse those regular expressions, here’s a summary:

There are four tests. If you fail 3 or more, you get 3.5 points, if you fail 4 you get another 3.5 points (my email failed all 4).
Here is how I failed them:
- The from address can’t have a bunch of words, including “project.” My from address includes my organization’s name: The Progressive Technology Project.
- The subject line cannot include a number of strings, including “last chance.” My subject line was “Last change to register for our webinar.”
- The subject line cannot include a number of other strings, including “webinar” (and also webcast and even strategy). My subject line was “Last chance to register for our webinar.”
- The body of the message cannot include a bunch of strings, including “register now.” Well, you won’t be suprised to know that my email contained the string “Register now.”

Hm. I’m glad I can now fix our email, but this doesn’t work so well for people with a name that includes “project” that like to organize webinars for which you have to register.

LVM Cache Surprises

Thu, 03 Mar 2022 08:27:10 -0400

By far the biggest LVM Cache surprise is just how well it works.

Between 2010 and 2020, my single, biggest and most consistent headache managing servers at May First has been disk i/o. We run a number of physical hosts with encrypted disks, with each providing a dozen or so sundry KVM guests. And they consume a lot of disk i/o.

This problem kept me awake at night and made me want to put my head on the table and cry during the day as I monitored the output of vmstat 1 and watched each disk i/o death spiral unfold.

We tried everything. Turned off fsck’s, turned off RAID monthly checks. Switched to less intensive backup systems. Added solid state drives and tried to stragically distribute them to our database partitions and other read/write heavy services. Added tmpfs file systems where it was possible.

But, the sad truth was: we simply did not have the resources to pay for the infrastructure that could support the disk i/o our services demanded.

Then, we discovered LVM caching (cue Hallelujah). We starting provisioning SSD partitions to back up our busiest spinning disk logical volumes and presto. Ten years of agony gone like a poof of smoke!

I don’t know which individuals are responsible for writing the LVM caching code but if you see this: THANK YOU! Your contributions to the world are noticed, appreciated and have had an enormous impact on at least one individual.

Some surprises

Filters

For the last two years, with the exception of one little heart attack, LVM caches have gone very smoothly.

Then, last week we upgraded 13 physical servers straight through from stretch to bullseye.

It went relatively smoothly for the first half of our servers (the old ones hosting fewer resources). But, after rebooting our first server with lvm caching going on, we noticed that the cached disk wasn’t accessible.

No problem, we reasoned. We’ll just uncache it. Except that didn’t work either. We tried every argument we could find on the Internet but lvm insisted that the block device from the SSD volume group (that provides the caching device) was not available. Running pvs showed an “unknown” device and vgs reported similar errors. Now I started to panic a bit. There was a clean shutdown of the server, so surely all the data had been flushed to the disk. But, how can we get that data? We started a restore from backup process because we really thought that data was gone for ever.

Then we had a really great theory: the caching logical volume comes from the SSD volume group, which gets decrypted after the spinning disk volume group.

Maybe there’s a timing issue? When the spinning disk volume group comes online, the caching logical volume is not yet available.

So, we booted into busybox, and manually decrypted the SSD volume first, followed by the spinning disk volume. Alas, no dice.

Now that we were fully desperate, we decided to restore the lvm configuration file for the entire spinning disk volume group. This felt kinda risky since we might be damaging all the currently working logical volumes, but it seemed like the only option we had.

The main problem was that busybox didn’t seem to have the lvm config tool we needed to restore the configuration from our backup (I think it might be there but it was late and we couldn’t figure it out). And, our only readily available live install media was a Debian stretch disk via debirf.

Debian stretch is pretty old and we really would have preferred to have the most modern tools available, but we decided to go with what we had.

And, that was a good thing, because as soon as we booted into stretch and decrypted the disks, the lvm volume suddenly appeared, happy as ever. We uncached it and booted into the host system and there it was.

We went to bed confused but relieved.

The next morning my co-worker figured it out: filtering.

During the stretch days we occassionally ran into an annoying problem: the logical volumes from guests would suddenly pop up on the host. This was mostly annoying but also it made possible some serious mistakes if you accidentally took a volume from a guest and used it on the host.

The LVM folks seemed to have noticed this problem and introduced a new default filter that tries to only show you the devices that you should be seeing.

Unfortunately for us, this new filter removed logical volumes from the list of available physical volumes. That does make sense for most people. But, not for us. It sounds a bit weird, but our setup looks like this:

One volume group derived from the spinning disks
One volume group derived from the SSD disks

Then we carve out logical volumes from each for each guest.

Once we discovered LVM caching, we carved out SSD logical volumes to be used as caches for the spinning logical volumes.

In restrospect, if we could start over, we would probably do it differently.

In any event, once we discovered the problem, we used the handy configuration options in lvm.conf to tweak the filters to include our cache disks and once again, everything is back to working.

Saturated SSDs

The other surprise seems unrelated to the upgrade. We have a phsyical server that has been suffering from disk i/o problems despite our use of LVM caching.

Our answer, of course, was to add more LVM caches to the spinning logical volumes that seemed to be suffering.

But somehow this was making things even worse.

Then, we finally just removed the LVM caches from all the spinning disks and presto, disk i/o problems seemed to go away. What? Isn’t that the opposite of what’s supposed to happen?

We’re still trying to figure this one out, but it seems that our SSDs are saturated, in which case adding them as a caching volume really is going to make things worse.

We’re still not sure why they are saturated when none of the SSDs on our other hosts are saturated, but a few theories include:

They are doing more writing and/or it’s a different kind of writing. I’m still not sure I quite have the right tool to compare this host with other hosts. And, this host is our only MySQL network database server, hosting hundreds of GBs of database - all writing/reading direclty onto the SSDs.
They are broken or substanard SSDs (smartctl doesn’t uncover any problems but maybe it’s a bad model?)

I’ll update this post as we learn more but welcome any suggestions in the comments.

Update: 2022-03-07

Two more possible causes:

Our use of the write back feature: LVM cache has a nice feature that caches writes to smooth out writes to the underlying disk. Maybe our disks are simply writing more then can be handled and not using write back is our solution. This server supports a guest with an unusually large disk.
Maybe we haven’t allocated a big enough LVM cache for the given volume so the contents are constantly being ejected?

Update: 2022-06-20

We figured it out shortly after my last update, but I never came back to update this post. After turning off lvm caching entirely, everything went back to normal during regular working hours (phew!).

But… the backup took close to 9 hours to complete. Comparable servers take a couple hours to backup. So, we started a process of backing up only half the top level directories. If the backup went through normally, we added back in half the directories we previously omitted.

Over the course of about 7 days we narrowed it down to just one top level directory and, after perusing that directory I found a 20GB file. That’s not terribly unusual and certainly should not be causing this level of crisis (we use an incremental backup system so only the parts of that file that have changed should get backed up).

But… this was no ordinary 20GB file. It was the file written to by a WordPress site with debug enabled. And, most importantly, it had been writing debug errors to this file for close to five years. That means two important things for us:

It most likely was fragmented all over the disk
It changed every day

From the backup perspective, it means every time the backup ran, the entire file had to be read to see what changed, which required pulling 20GB in the most inefficient way possible causing a major spike in disk i/o.

From the LVM cache perspective, it means that every time this file was written to (i.e. every time this lousy WordPress site logged a debug message), the file had to be read from the spinning disk into the ssd cache disk. I imagine that lvm cache doesn’t have a “only load what’s changed” feature and instead simply re-reads the entire file everytime it changes.

Mystery solved. LVM cache still rocks. And now we have a new DDOS vector for very patient people.

Putty Problems

Mon, 20 Sep 2021 08:27:10 -0400

I upgraded my first servers from buster to bullseye over the weekend and it went very smoothly, so big thank you to all the debian developers who contributed your labor to the bullseye release!

This morning, however, I hit a snag when the first windows users tried to login. It seems like a putty bug (see update below).

First, the user received an error related to algorithm selection. I didn’t record the exact error and simply suggested that the user upgrade.

Once the user was running the latest version of putty (0.76), they received a new error:

Server refused public-key signature despite accepting key!

I turned up debugging on the server and recorded:

Sep 20 13:10:32 container001 sshd[1647842]: Accepted key RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4 found at /home/XXXXXXXXX/.ssh/authorized_keys:6
Sep 20 13:10:32 container001 sshd[1647842]: debug1: restore_uid: 0/0
Sep 20 13:10:32 container001 sshd[1647842]: Postponed publickey for XXXXXXXXX from xxx.xxx.xxx.xxx port 63579 ssh2 [preauth]
Sep 20 13:10:33 container001 sshd[1647842]: debug1: userauth-request for user XXXXXXXXX service ssh-connection method publickey [preauth]
Sep 20 13:10:33 container001 sshd[1647842]: debug1: attempt 2 failures 0 [preauth]
Sep 20 13:10:33 container001 sshd[1647842]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
Sep 20 13:10:33 container001 sshd[1647842]: debug1: trying public key file /home/XXXXXXXXX/.ssh/authorized_keys
Sep 20 13:10:33 container001 sshd[1647842]: debug1: fd 5 clearing O_NONBLOCK
Sep 20 13:10:33 container001 sshd[1647842]: debug1: /home/XXXXXXXXX/.ssh/authorized_keys:6: matching key found: RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4
Sep 20 13:10:33 container001 sshd[1647842]: debug1: /home/XXXXXXXXX/.ssh/authorized_keys:6: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
Sep 20 13:10:33 container001 sshd[1647842]: Accepted key RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4 found at /home/XXXXXXXXX/.ssh/authorized_keys:6
Sep 20 13:10:33 container001 sshd[1647842]: debug1: restore_uid: 0/0
Sep 20 13:10:33 container001 sshd[1647842]: debug1: auth_activate_options: setting new authentication options
Sep 20 13:10:33 container001 sshd[1647842]: Failed publickey for XXXXXXXXX from xxx.xxx.xxx.xxx port 63579 ssh2: RSA SHA256:t3DVS5wZmO7DVwqFc41AvwgS5gx1jDWnR89apGmFpf4
Sep 20 13:10:39 container001 sshd[1647514]: debug1: Forked child 1648153.
Sep 20 13:10:39 container001 sshd[1648153]: debug1: Set /proc/self/oom_score_adj to 0
Sep 20 13:10:39 container001 sshd[1648153]: debug1: rexec start in 5 out 5 newsock 5 pipe 8 sock 9
Sep 20 13:10:39 container001 sshd[1648153]: debug1: inetd sockets after dupping: 4, 4

The server log seems to agree with the client returned message: first the key was accepted, then it was refused.

We re-generated a new key. We turned off the windows firewall. We deleted all the putty settings via the windows registry and re-set them from scratch.

Nothing seemed to work. Then, another windows user reported no problem (and that user was running putty version 0.74). So the first user downgraded to 0.74 and everything worked fine.

Update

Wow, very impressed with the responsiveness of putty devs!

And, who knew that putty is available in debian??

Long story short: putty version 0.76 works on linux and, from what I can tell, works for everyone except my one user. Maybe it’s their provider doing some filtering? Maybe a nuance to their version of Windows?

Anyone still using gitweb?

Wed, 18 Aug 2021 08:27:10 -0400

It seems like the self-hosting git world has all moved to gitlab or gitea.

For a number of reasons not worth enumerating, I’m still running gitolite and recently decided I wanted to checkout my code via https using gitweb.

I got through most of the installation and configuration without trouble (I could browse via the web and see all my repositories). But, when I tried to git clone using the https address I got a fatal “not found” error.

It seems that gitweb, out of the box, allows for easy web-browsing of git repositories but needs some extra work if you want to clone over https. Specifically, you need to use git-http-backend.

The git-http-backend man page is very useful, but assumes you are accessing your repos via https://example.org/git instead of simply https://git.exmple.org.

These lines are my variation to the suggested apache configuration lines provided by man git-http-backend.

They differ by:

allowing for web access without specifying a subdirectory
using the debian /usr/lib/git-core path instead of /usr/libexec/git-core
removing git-receive-pack since I only plan to clone and don’t plan to push back to this repo.

DocumentRoot /usr/share/gitweb
<Directory /usr/share/gitweb>
  Options +FollowSymLinks +ExecCGI
  AddHandler cgi-script .cgi
  Require all granted
</Directory>
<Directory /usr/lib/git-core>
  Require all granted
</Directory>
SetEnv GIT_PROJECT_ROOT /var/lib/git
AliasMatch ^/(.*/objects/[0-9a-f]{2}/[0-9a-f]{38})$          /var/lib/git/$1
AliasMatch ^/(.*/objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/lib/git/$1
Alias /static /usr/share/gitweb/static
ScriptAliasMatch \
  "(?x)^/(.*/(HEAD | \
    info/refs | \
    objects/info/[^/]+ | \
    git-upload-pack))$" \
  /usr/lib/git-core/git-http-backend/$1
ScriptAlias / /usr/share/gitweb/gitweb.cgi/

The main trick is to direct some requests to apache2, some requests to /usr/lib/git-core/git-http-backend, and everything else to gitweb.cgi.

Fixing old PHP code

Sat, 31 Jul 2021 11:11:05 -0400

I wrote a control panel in 2005 using PHP, without any framework. Who could have guessed it would still be in production now?

We’ve recently decided to put off replacing it for a few years, so I have to fix all the deprecation warnings, which are almost all due to:

while(list($k, $v) = each($array)) {

At some point, early in my PHP coding life, someone told me foreach($array as $k => $v) { was bad. I don’t even remember why. But it stuck, so my code is littered with the while/list/each approach. If I ever wrote malware in PHP you could definitely fingerprint me with this one.

I’m working on some sed magic to fix them, starting with:

find . -name '*.php' -exec sed -E -i 's#while\(list\((\$[a-z_]*), ?(\$[a-z_]*)\) = each\((\$[a-z_>-]+)\)\) \{#foreach(\3 as \1 => \2) {#g' '{}' \;

But… it misses this variation:

while(list(, $v) = each($array)) {

So I also ran:

find . -name '*.php' -exec sed -E -i 's#while\(list\(,(\$[a-z_]*)\) = each\((\$[a-z_>-]+)\)\) \{#foreach(\2 as \1) {#g' '{}' \;

I ended up with 10 replacments I had to do by hand (while(list($k) = each($array)) and a few others with unusual spacing).

Google and Bitly

Sun, 18 Jul 2021 13:25:59 -0400

It seems I’m the only person on the Internet who didn’t know sending email to Google with bit.ly links will tank your deliverability. To my credit, I’ve been answering deliverability support questions for 16 years and this has never come up.

Until last week.

For some reason, at May First we suddenly had about three percent of our email to Google deferred with the ominous sounding:

“Our system has detected that this message is 421-4.7.0 suspicious due to the nature of the content and/or the links within.”

The quantity of email that accounts for just three percent of mail to Google is high, and caused all kinds of monitoring alarms to go off, putting us into a bit of panic.

Eventually we realized all but one of the email messages had bit.ly links.

I’m still not sure whether this issue was caused by a weird and coincidental spike in users sending bit.ly links to Google. Or whether some subtle change in the Google algorithm is responsible. Or some change in our IP address reputation placed greater emphasis on bit.ly links.

In the end it doesn’t really matter - the real point is that until we disrupt this growing monopoly we will all be at the mercy of Google and their algorithms for email deliverability (and much, much more).

From Ikiwiki to Hugo

Fri, 16 Jul 2021 08:27:10 -0400

Back in the days of Etch, I converted this blog from Drupal to ikiwiki. I remember being very excited about this brand new concept of static web sites derived from content stored in a version control system.

And now over a decade later I’ve moved to hugo.

I feel some loyalty to ikiwiki and Joey Hess for opening my eyes to the static web site concept. But ultimately I grew tired of splitting my time and energy between learning ikiwiki and hugo, which has been my tool of choice for new projects. When I started getting strange emails that I suspect had something to do with spammers filling out ikiwiki’s commenting registration system, I choose to invest my time in switching to hugo over debugging and really understanding how ikiwiki handles user registration.

I carefully reviewed anarcat’s blog on converting from ikiwiki to hugo and learned about a lot of ikiwiki features I am not using. Wow, it’s times like these that I’m glad I keep it really simple. Based on the various ikiwiki2hugo python scripts I studied, I eventually wrote a far simpler one tailored to my needs.

Also, in what could only be called a desperate act of procrastination combined with a touch of self-hatred (it’s been a rough week) I rejected all the commenting options available to me and choose to implement my own in PHP.

What?!?! Why would anyone do such a thing?

I refer you to my previous sentence about desperate procrastination. And also… I know it’s fashionable to hate PHP, but honestly as the first programming language I learned, there is something comforting and familiar about it. And, on a more objective level, I can deploy it easily to just about any hosting provider in the world. I don’t have to maintain a unicorn service or a nodejs service and make special configuration entries in my web configuration. All I have to do is upload the php files and I’m done.

Well, I’m sure I’ll regret this decision.

Special thanks to Alexander Bilz for the anatole hugo theme. I choose it via a nearly random click to avoid the rabbit hole of choosing a theme. And, by luck, it has turned out quite well. I only had to override the commento partial theme page to hijack it for my own commenting system’s use.

How to Meet Online with Simultaneous Interpretation

Tue, 22 Jun 2021 09:14:50 +0000

May First Movement Technology has been running a public Jitsi Meet instance since well before the pandemic to support Internet-based, video meetings for folks who don’t want to rely on corporate and proprietary infrastructure.

However (until this week - see below), we haven’t been using it for our own meetings for one main reason: simultaneous interpretation. We’re an international organization with roots in the US and Mexico and we are committed to building a bi-national leadership with a movement strategy that recongizes the symbolic and practical disaster of the US/Mexico border.

As a result, we simply can’t hold a meeting without simultaneous interpretation between english and spanish.

Up to now, we’ve worked out a creative way to have mumble meetings with simultaneous interpretation. In short: we have a room for interpretation. If you move into the interpretation room, you hear the interpreter. If you move into the main room, you hear the live voices of the participants. You can switch between rooms as needed. This approach is rock solid, and we benefit from mumble’s excellent performance in low bandwidth situations and the availability of mumble clients on both Android and iPhones.

However, there are limitations, which include:

You can’t hear both the live voices and the interpretation at the same time: it’s one or the other. If you are in a face-to-face meeting and receiving interpretation via headphones, you can see the person talking and even remove the headphones from one ear to get a sense of the tone and emotions of the speaker. Not with mumble. In fact, you can’t even tell who is speaking.
Two chat rooms: If you chat to the live group, it’s only seen by the live group. If you chat with the interpretation group, it’s only seen by the interpretation group.
No video in mumble: well, some people consider this a positive. I’ll leave it at that.

After years of reviewing the many dead-end threads and issue requests around simultaneous interpretation on the Jitsi boards (and the Big Blue Button boards for that matter) I finally came across the thread that led to the pull request that changed everything.

With the ability to control local volume via the Jitsi Meet API, I was able to pull together a very small amount of code to produce Jitsi Simultaneous Interpretation (JSI) - a way to run your Jitsi Meet server with an interpretation slider at the top allowing you to set the volume of the interpreter at any time during the meeting.

It’s still not perfect - the main problem is that you can’t use any of the Jitsi Meet apps - so it runs well on most desktops, but when it comes to cell phones, it only runs (in browser) on modern android phones.

From openbox to sway

Tue, 11 May 2021 08:33:26 +0000

I’ve been running the Openbox window manager since 2005. That’s longer then I’ve lived in any one apartment in my entire life!

However, over the years I’ve been bracing for a change.

It seems clear the Wayland is the future, although when that future is supposed to begin is much more hazy.

Really, I’ve felt a bit like a ping pong ball, from panicking over whether Xorg is abandoned (with a follow up from a X server maintainer) to anxiously wondering if literally everything will break the moment I switch to Wayland.

In fact, I started this blog post over a year ago when I first decided to switch from the Openbox to Sway.

This is my third major attempt to make the change and I think it will finally stick this time.

In retrospect, it would have been more sensible to first switch from openbox to i3 (which is a huge transition) and then from i3 to sway, but I decided to dive into the deep end with both changes. Note: thanks to a helpful comment on this blog, I learned that there is waybox, an openbox clone for wayland, which would have been another version of a less drastic change.

So… I’m on debian bullseye so I installed sway and friends (from sid).

Then I copied /etc/sway/config to ~/.config/sway/config.

I start openbox after logging in with exec startx so after rebooting, I ran exec sway and to my astonishment sway started. Hooray!

However, I found that ssh-agent wasn’t running so I couldn’t ssh into any servers. That’s kinda a problem.

Launching ssh-agent under openbox was buried deep in /etc/X11/Xsession.d/90x11-common_ssh-agent and clearly was not going to happen via wayland.

Since programs using ssh-agent depend on the environment variables SSH_AUTH_SOCK and SSH_AGENT_PID being globally available I thought I could simply run $(eval ssh-agent) via my tty terminal before running exec sway.

And, that would have worked. Except… I like to add my keys via ssh-add -c so that everytime my key is being used I get a ssh-askpass prompt to confirm the use.

It seems that since ssh-add is started before a window manager is running, it can’t run the prompt.

Ok, we can fix this. After searching the web, I came upon a solution of running ssh-agent via systemctl --user:

# This service myst be started manually after sway
# starts.
[Unit]

Description=OpenSSH private key agent
IgnoreOnIsolate=true

[Service]
Type=forking
Environment=SSH_AUTH_SOCK=%t/ssh-agent.socket
ExecStart=/usr/bin/ssh-agent -a $SSH_AUTH_SOCK

Then, in my ~/.bashrc file I have:

if [ -n WAYLAND_DISPLAY ]; then
  export SSH_AUTH_SOCK=/run/user/1000/ssh-agent.socket
fi

I think $SSH_AGENT_PID is only used by ssh-agent to kill itself. Now that is running via systemd - killing it should be do-able without a global environment variable. Done? Hardly.

I’ve been using impass (nee assword) happily for years but alas it is tightly integrated with xdo and xclip.

So… I’ve switched to keepassxc which works out of the box with wayland.

My next challenge was the status bar. Farewell faithful tint2. One of the reasons I failed on my first two attempts to switch to Sway was the difficulty of getting the swaybar to work how I wanted, particularly with nm-applet. Two things allowed me to move forward:

waybar was added to Debian. Thank you Debian waybar maintainers!
I gave up on having nm-applet work the way I’m used to working and resigned myself to using nmtui. Sigh.

Next up: the waybar clock module doesn’t work, but that is easy enough to work around.

Replacing my uses of xclip with wl-clipboard was a little tedious but really not that difficult.

Getting my screen shot and screen recorder functionality was a bit harder. I did a lot of searching before I finally found and compiled both swappy, screen shot and wf-recorder.

In the course of all my adventures, I came across the following helpful tips:

Updates

I’ve installed libreoffice-gtk3 to ensure libre office runs under wayland
I’ve installed the latest Gimp via flatpak to get proper wayland support. Update: as of 2022-03-29, it seems that Gimp 2.10.30 in Debian Bullseye works fine with Sway.
I’ve exported MOZ_ENABLE_WAYLAND to ensure firefox works properly.
I’ve found that passing -c to my ssh-add command to ensure I am prompted for each use of my key seems to cause sway to crash intermittently. Update: This is no longer the case. Not sure why.
I learned about a work around to get screen sharing to work in zoom. Somehow this actually works. Amazing. Unfortunately, though, sharing your screen in the context of a Zoom meeting pins your screen for all participants. So, sharing your desktop through your camera really doesn’t cut it. I finally landed an an obvious work around: Install chromium (which runs under X11); Install the the chrome zoom redirector extension (update: This extension is no longer available); Open zoom links in chromium; You can now share other chromium windows. Not the full desktop or any wayland window, but if you only need to share a web browser window, you are set; For the record, Zoom links normally are in the format: https://us04web.zoom.us/j/123456. If you want to force the use of the web client, just change the “j” to “wc”: https://us04web.zoom.us/wc/123456.
Speaking of screen sharing - when using Firefox, I can only share Xwayland screens. Firefox is running under wayland so I can’t share it. Chromium is running under xwayland, so I have to use Chromium when screen sharing.
Wait, scratch that about screen sharing in Firefox. I’ve installed xdg-desktop-portal-wlr, added export XDG_CURRENT_DESKTOP=sway and export XDG_SESSION_TYPE=wayland to my .bashrc, and after hours of frustration, realize that I needed to configured firejail to allow it so that I can share my entire screen in Firefox. It doesn’t yet support sharing a specific window, so I still have to keep chromium around for that (and Chromium can only share xwayland windows). Sigh. Oh, one more thing about Firefox: the option to choose what to share doesn’t have “Entire Screen” as an option, you are just supposed to know that you should choose “Use operating system settings”.
I still am getting weekly crashes. Some of them I’ve fixed by switching to wayland friendly versions (e.g. Libre Office and Gimp) but others I haven’t yet tracked down. Update: No longer getting weekly crashes. Very stable.
My keyboard does not have an altgr key, so even though I have selected the “English (US) - English (intl., with AltGr dead keys)” I can’t get accent marks. I went down a rabbit hole of trying to re-map the Alt key to the right of my space bar but it all seemed too complicated. So - I found a way easier approach. In my ~/.config/sway/config file I have: bindsym Mod4+e exec wtype "é". I have repeated that line for the main accent marks I need.
Due to a Firefox Bug, when I share my desktop or mic or camera, the sharing indicator expands like a normal tiling window instead of remaining a tiny little box on each desktop reminding me that I’m sharing something. I’d prefer to have it be a tiny little box, but since I can’t figure that out, I’ve disabled it by typing about:config in the Firefox location window, searching for privacy.webrtc.legacyGlobalIndicator and setting it to False. The reddit thread also suggested finding privacy.webrtc.hideGlobalIndicator and setting it to True, but that setting doesn’t seem to be available and setting the first one alone seems to do the trick.
Oh, one more environment variable to set: GDK_BACKEND=wayland,x11. First I just set it to wayland to get gtk3 apps to use wayland (like gajim). But that broke electron apps (like signal) which notice that variable but don’t have a way to display via wayland (at least not yet). Setting it to “wayland,x11” shows the priority. Thank you ubuntu community.
I’ve also finally consolidated where my environment variables go. I’ve added them all to ~/.config/sway/env. That seems like an official sway place to put them, but sway doesn’t pay any attention to them. So I start sway via my own bash script which sources that file via [ -f "$HOME/.config/sway/env" ] && . "$HOME/.config/sway/env" before exec‘ing sway.

The problem with Richard Stallman is not about free speech

Mon, 29 Mar 2021 08:50:40 +0000

Free speech and censorship are critically important issues. And, using them to defend Richard Stallman’s return to the Free Software Foundation (FSF) board is just plain wrong.

Richard Stallman resigned from the Board in 2019 after he sent an email in defense of Marvin Minsky (Minsky is accused of raping one of Jeffreys Epstein’s victims).

Stallman’s fateful email, however, is just one piece of the reason for why he should not be on the board. The full story is about his history of abuse toward women and is extensive.

On March 21st, 2021, Stallman announced he is back on the board.

There are profound reasons why any movement interested in equitable and open participation would want to publicly distance themselves from Stallman. However, the long form defenses of Stallman, including a note from Nadine Strossen, the former executive director of the ACLU, quoted in this defense, persist.

Many of the arguments defending Richard Stallman (including the one from Strossen) are grounded in a belief that Stallman is being punished for his unpopular political views, which deserve to be defended on the grounds of freedom of expression.

That’s wrong.

Stallman should be kicked off the board because he has a long history of abusing his position to hit on women, which, when combined with his public opinions on under-age sex and his defense of Minsky, send a strong signal that the FSF does not care about the participation of women.

Being on a board of directors is a privilege, not a right. Being removed from a board is not a punnishment. And being criticized and removed from a board because your behavior and public statements are an obstacle to building an inclusive and equitable movement is what every board should strive to do.

If we are going to make this an issue about free expression, it should be about all the political expression lost to the free software movement because Stallman’s unequal behavior toward women excluded an enormous number of talented individuals.

So... is Signal good or bad?

Fri, 29 Jan 2021 15:37:11 +0000

After Facebook updated their Whatsapp privacy policy, and a certain rich capitalist who doesn’t like Facebook for reasons different then mine told the world to use Signal, Signal’s downloads went up by 4,200%.

As often happens when something becomes popular, the criticisms start to fly!

For the record, I currently think promoting Signal is an important tactical strategy for the left. [I also think we should promote and install federated chat apps like conversations and element and delta chat whereever it is possible.]

Here are some of the main criticisms I hear that I think are a distraction:

Signal forces you to use the Google Play store and Google Services: This isn’t true any more. You can download the apk directly on a phone without any Google services and it works great. The app will alert you to new versions.

Don’t get me wrong: the Signal network still depends on Google services. And, we should be avoiding corporate technology and building our own infrastructure. However, in practice, Signal is an alternative to Whatsapp and Telegram - which not only use the same corporate services but are proprietary technology that is fully owned by powerful tech giants. Signal is still a non-profit organization with a vastly different mission.
Signal’s approach to privacy isn’t perfect (the most common variation on this theme is that a state actor could monitor your outgoing communications and the incoming communications of the person you are communicating with and prove that you are communicating with each other).

This criticism missed what makes Signal so important. The beauty of Signal is that it addresses the “woops!” moment most privacy activists had when Snowden’s data trove become public: it provides mass privacy to stop mass surveillance. Prior to 2013, most tech/privacy activists were focused on the “targeted” individual approach to privacy, working hard to make sure our tools were as absolutely perfect as possible for the tiny percentage of people who know they are under surveillance. Very little effort went into getting them adopted on a mass scale.

Criticizing Signal for not providing perfect privacy misses that fact that these things often are trade offs.

This trade-off also applies to the first point - dependency on Google services makes installation far easier for suporting millions of people.

Here are some criticisms that I think are nuanced:

Signal is a centralized app: this criticism often includes examples of Moxie (Signal’s founder) refusing and actively discouraging attempts by others to write software that interacts with Signal.

Signal is free software, which is a major improvement over most corporate technology. But since it’s entirely controlled by one entity, it can be shutdown in a heart beat. And, if Signal changes direction, we cannot easily take the work we have all invested in learning signal and create our own version that reflects our values.

This problem is in contrast to federated systems like email - where anyone can run their own email server and apply their own policies. If one email serveer is shutdown, you can move to another.

I agree with this critique, but I think it’s nuanced because of the trade offs. Having full control over the entire network and all the software provides a level of reliability and consistency that would not be possible with a federated protocol. And, we already have three different, fully viable federated chat protocols (see above). I’d rather have Signal be Signal and invest our energy on a federated chat system via the existing, well-developed alternatives.

This opinion is tactical - and could change at any moment. I think there will come a time when we are going to tell the world to move from Signal to the best available federated protocol. But I’m not convinced we have a robust enough federated chat infrastructure to support that move.
Signal forces you to use your phone number as an identifier: You can’t get a Signal account without a phone number. And you generally can’t get a phone number without revealing some aspect of your identity. That makes staying anoymous very difficult. There are reports of a new Signal feature making it possible to avoid revealing your phone number when communicating with others, but you would still need a phone number to get an account because a SMS or phone call confirmation is required.
Signal isn’t getting ahead of the curve on abuse: There’s an interesting piece informed by former Signal staff people about the management’s resistance to getting ahead of the curve when it comes to abuse. How would signal respond to reports of harrassment? What would signal do if it recognized facsists movements organizing on its platform? Any mass platform that is not planning for abuse is going to be in big trouble very soon.

These last two are not exactly two sides of the same coin, but they are related. How Signal manages to balance privacy and protection from abuse will be the real test as to whether promoting Signal continues to be a useful strategy for the left.

Being your own Certificate Authority

Sun, 15 Nov 2020 13:48:45 +0000

There are many blogs and tutorials with nice shortcuts providing the necessary openssl commands to create and sign x509 certficates.

However, there is precious few instructions for how to easily create your own certificate authority.

You probably never want to do this in a production environment, but in a development environment it will make your life signficantly easier.

Create the certificate authority

Prepare your CA directories

Pick a directory to store your keys and certificates in.

Then make a subdirectory for the certficiate authority and some required directories:

mkdir ca
mkdir ca/private
touch index.txt

Create the key and certificate

Then, make your certificate authority key and certificate:

openssl genrsa -out ca/private/cakey.pem 2048
openssl req -x509 -new -nodes -key ca/private/cakey.pem -sha256 -days 1024 -out ca/cacert.pem

Some tips:

You will be prompted to enter some information about your certificate authoirty. Provide the minimum information - i.e., only overwrite the defaults. So, provide a value for Country, State or Province, and Organization Name and leave the rest blank.
You probably want to leave the password blank if this is a development/testing environment.

Want to review what you created?

openssl x509 -text -noout -in ca/cacert.pem

Prepare your openssl.cnf file

With Debian Trixie, the default openssl configuration file seems to be /usr/lib/ssl/openssl.cnf. Prior versions used /etc/ssl/openssl.cnf.

In this document we will explicitly use /etc/ssl/openssl.cnf to get the Debiand defaults.

But, there are two bits missing from the Debian configuration that have to be added.

At the very top add: subjectAltName = ''.
In the [user_cert] section, add: subjectAltName=$ENV::subjectAltName

Create a your first key and ceritificate signing request

First, pick your domain names (aka “common” names). For example, example.org and www.example.org.

Set those values in an environment variable. If you just have one:

export subjectAltName=DNS:example.org

If you have more then one:

export subjectAltName=DNS:example.org,DNS:www.example.org

If you have a wild card domain:

export subjectAltName='DNS:*.example.org'

Next, create a key and a certificate signing request:

openssl req -config /etc/ssl/openssl.cnf -new -nodes -addext "$subjectAltName" -out new.csr -keyout new.key

Again, you will be prompted for some values (country, state, etc) - be sure to choose the same values you used with your certficiate authority! I honestly don’t understand why this is necessary (when I set different values I get an error on the signing request step below). Maybe someone can add a comment to this post explaining why these values have to match?

Also, you must provide a common name for your certificate - you can choose the same name as the altSubjectNames value you set above (but just one domain).

Want to review what you created?

openssl req -config /etc/ssl/openssl.cnf -in new.csr -text -noout

Sign it!

At last the momenet we have been waiting for:

openssl ca -config /etc/ssl/openssl.cnf -keyfile ca/private/cakey.pem -cert ca/cacert.pem -out new.crt -outdir . -rand_serial -infiles new.csr

Now you have a new.crt and new.csr that you can install via your web browser, mail server, etc specification.

Smoke Test

This command will confirm that the certificate is trusted by your certificate authority.

openssl verify -config /etc/ssl/openssl.cnf -no-CApath -CAfile ca/cacert.pem new.crt

But wait, there’s still a question of trust

You probably want to tell your computer or browser that you want to trust your certificate signing authority.

Command line tools

Most tools in linux by default will trust all the certificates in /etc/ssl/certs/ca-certificates.crt. (If that file doesn’t exist, try installing the ca-certificates package). If you want to add your certificate to that file:

cp cacert.pem /usr/local/share/ca-certificates/cacert.crt
sudo dpkg-reconfigure ca-certificates

Want to know what’s funny? Ok, not really funny. If the certificate name ends with .pem the command above won’t work. Seriously.

Once your certificate is installed with your web server you can now test to make sure it’s all working with:

gnutls-cli --print-cert $domainName

Want a second opinion?

curl https://$domainName
wget https://$domainName -O-

Both will report errors if the certificate can’t be verified by a system certificate.

If you really want to narrow down the cause of error (maybe reconfiguring ca-certificates didn’t work)?

curl --cacert /path/to/your/cacert.pem --capath /tmp

Those arguments tell curl to use your certificate authority file and not to load any other certificate authority files (well, unless you have some installed in the temp directory).

Web browsers

Firefox and Chrome have their own store of trusted certificates - you’ll have to import your cacert.pem file into each browser that you want to trust your key.

Renewing

In the first step, you created a certificate signing authority key with an expiration of 1,024 days.

With luck, you’ll still be using it after 3 years which means you’ll need to renew it.

Start by changing into the ca directory.

Then, create a new certificate signing request:

openssl x509 -x509toreq -in ca/cacert.pem -signkey ca/private/cakey.pem -out new-server.csr

That command creates the file new-server.csr - a certificate signing request.

Now, simply sign it:

openssl x509 -req -days 1024 -in new-server.csr -signkey private/cakey.pem -out new-cacert.pem

This command generates your brand new cacert.pem file, but with the new name new-cacert.pem.

Now, you simply use the new file to replace your old cacert.pem file.

LVM Cache woops

Thu, 23 Apr 2020 13:51:25 +0000

At May First, disk i/o has been our most serious bottle neck for many years. We have plenty of RAM, disk space and even CPU.

But when too much data is being written to our spinning disks everyting grinds to a halt.

As we have been adding SSD disks to our servers, we’ve recently begun experimenting with adding SSD-backed lvm caches. This approach has had a tremendous impact - resolving most of our disk i/o problems.

However, this morning we rebooted one of those virtual guests and I almost had a heart attack:

0 claudette:~# mount /home
mount: special device /dev/mapper/vg_claudette0-home does not exist
32 claudette:~# lvs vg_claudette0/home
  LV   VG            Attr       LSize   Pool             Origin       Data%  Meta%  Move Log Cpy%Sync Convert
  home vg_claudette0 Cwi---C--- 309.00g [home_cachepool] [home_corig]                                        
0 claudette:~# ls /dev/mapper/
control             vg_claudette0-swap_1  vg_claudette0-var
vg_claudette0-root  vg_claudette0-tmp     vg_claudette0-var+lib+mysql
0 claudette:~#

Wah! lvm ate our data!

Let’s remove the cache and return to the way it was:

0 claudette:~# lvconvert --uncache vg_claudette0/home
  /usr/sbin/cache_check: execvp failed: No such file or directory
  Check of pool vg_claudette0/home_cachepool failed (status:2). Manual repair required!
  Failed to active cache locally vg_claudette0/home.
5 claudette:~#

Wah! That doesn’t work either!

Let’s repair:

0 claudette:~# lvconvert --repair vg_claudette0/home_cachepool
  Using default stripesize 64.00 KiB.
  Operation not permitted on cache pool LV vg_claudette0/home_cachepool.
  Operations permitted on a cache pool LV are:
  --splitcache    (operates on cache LV)

5 claudette:~#

What is happening!!

I booted into a live rescue disk with a more modern version of lvm that really should support the --repair option:

0 debirf-rescue:~# lvconvert --repair vg_claudette0/home_cachepool
  /dev/vg_claudette0/lvol1: not found: device not cleared
  Aborting. Failed to wipe start of new LV.
  WARNING: If everything works, remove vg_claudette0/home_cachepool_meta0 volume.
  WARNING: Use pvmove command to move vg_claudette0/home_cachepool_cmeta on the best fitting PV.
0 debirf-rescue:~#

Help! Help!

Wait… thanks to a cool headed colleauge, it turns out the the only problem was that thin-provisioning-tools was not installed on the host.

Feel free to review the whole fiasco as it unfolded.

Free software in the age of Corona Virus

Fri, 27 Mar 2020 15:51:00 +0000

There are the printed remarks I made during the May First webinar on free software during the Corona Virus pandemic, which can be heard via the recording.

Hi everyone, so glad to be here with all of you during these unprecedented times.

I’d like to first make clear that I’m proud of the tools and services that May First and our movement in general has made available to everyone seeking a way to continue our important organizing work online. And everyone on this call is a testament to our movement’s ability to choose to build online meeting software that respects our privacy and invests in movement owned technology.

However, I’d like to address something we can’t avoid: capitalism has created online, full video Internet conferences with as many people as you want where you can see everyone’s faces and share any screen.

Our movement has not figured out how to provide that service in a way that protects our privacy and ensures such a vital tool remains in the control of the movement and not in the hands of the corporate world we are fighting so hard to stop.

As a result of this shortcoming, our movement is, as we speak, investing heavily - both in terms of money and in terms of learning new tools - to ensure that these corporate solutions become the default and entrenched methods for meeting online.

I’d like to address a couple questions: How did we get here? And what can we do about it?

We could fill an entire conference on the techno/political reasons why video on the Internet is hard, so I’ll just skim the surface: The success of the Internet is largely due to it’s open nature: there is a standard for how information can be shared, and anyone who builds a program that uses that standard can inter-operate with everyone else.

From the birth of the corporate Internet, capitalists have been trying to control the Internet in a way that ensures they will profit the most from it. And video, which had no standards defined for it when things really took off back in 1994, has been one of the biggest prizes. As a result, the last 20 years have witnessed battles over what standards will govern how video is transmitted over the Internet and, despite some recent victories, the ones promoting free and open tools have largely lost (those of you old enough to remember that abomination called “flash” will know what I’m talking about).

The result is a fractured Internet when it comes to video. While things have been recently changing, for most of the last 20 years it has been impossible to come up with a way to transmit video that will work on all computers and all cell phones. As a result, only organizations with a lot of capital have been able to afford the engineering teams needed to write the software for all the devices.

However there’s also a second reason, one that might be even more important: the movement has not prioritized it. Why? We certainly know how to prioritize Internet development.

In the late 1980’s the left wing Internet organization IGC organized a left-wing forum system for earlier Internet users. The Zapatistas prioritized the use of the Internet in the mid 1990’s before anyone had figured it out. The Global Justice movement build the global Indymedia Network when nobody knew how to publish our own news. In the early 2000’s CiviCRM created a database system that the movement needed and PTP has further developed it for the Movement as Powerbase.

However, when it comes to video conferencing we seem to have ceded this work to the corporatate world.

The next question is: how do we turn this around? That’s for us to figure out together, but I’ll start with some ideas:

Let’s design campaigns that demonstrate to the movement why this is important. No Tech for Ice is a great example - it shows how big tech has a real and direct impact on our issues. Same with anti-surveillance campaigns.
Let’s remember our values: it’s not just about video, it’s about ensuring that everyone has access, even in low bandwidth environments, where people who can’t make an international phone call can still connect, where people can speak the language they want and get interpreted in real time.
Let’s use our imaginations to change our culture: we can’t just move an “in person” meeting to an online format. In fact, maybe our in person formats weren’t working so well anyway! How do we build a movement culture that supports unifying our disperate issue focused groups into a powerful force for systemic change? And now, how does the Internet support that agenda?

Programming with 6 year olds

Fri, 27 Mar 2020 15:45:31 +0000

I know there is a pedagogically appropriate way to teach 6 year olds how to program and this is not it.

But, it was still fun. And maybe we programmers could use the bark sound more often?

#!/usr/bin/python3

import random
import time
import subprocess

while 1:
    answers = [
            "What is your question?",
            "There is no answer.",
            "53",
            "The answer to your question lies within your ear.",
            "Blue berries!",
            "The grasshopper is 20 feet tall.",
            "Elephants are the size of bacteria.",
            "True",
            "False",
            "Yes, no, well, maybe.",
            "I have no idea.",
            "I do not know WHAT you are talking about.",
            "That's rude.",
            "Ask your dad.",
            "I'm sick, please ask someone else.",
            "Maybe you should ask yourself that question.",
            "Could you ask a different question?",
            "You could probably find out by asking the Internet.",
            "Is that question really so important to you?",
            "Can I have a turn asking a question?",
            "How come you get to ask all the questions?",
            "Absolutely not.",
            "You are asking so many questions and haven't even introduced yourself to me.",
            "What other question do you have?",
            "What are you talking about?",
            "I couldn't possibly answer that question.",
            "Can we play a different game?",
            "That makes my brain hurt.",
            "Why are you asking me a question you know the answer to?",
            "Ha ha ha ha ha ha ha ha.",
            "Boo boo butt",
            "Really, ask someone else, I am sick!",
            "Computer, stop talking - you are interrupting me",
            "You are really bad at typing",
            "Why do you keep asking me questions?",
            "Ask someone else already.",
            "The answer is.",
            "You have a bug in your programming."
    ]
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    question = "What is your question? "
    subprocess.call(["espeak", question])
    user_question = input(question)
    subprocess.call(["espeak", "Your question is ... " + user_question])
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])

    answer = random.choice(answers)

    print("And the answer is...")
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    subprocess.call(["espeak", "And the answer is"])
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])

    time.sleep(2)

    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    print("... the answer is....")
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    subprocess.call(["espeak", "the answer is"])

    time.sleep(2)

    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])
    print(answer)
    subprocess.call(["espeak", answer])
    subprocess.call(["paplay", "/usr/share/sounds/gnome/default/alerts/bark.ogg"])

Editing video without a GUI? Really?

Tue, 08 Oct 2019 09:21:39 +0000

It seems counter intuitive - if ever there was a program in need of a graphical user interface, it’s a non-linear video editing program.

However, as part of the May First board elections, I discovered otherwise.

We asked each board candidate to submit a 1 - 2 minute video introduction about why they want to be on the board. My job was to connect them all into a single video.

I had an unrealistic thought that I could find some simple tool that could concatenate them all together (like mkvmerge) but I soon realized that this approach requires that everyone use the exact same format, codec, bit rate, sample rate and blah blah blah.

I soon realized that I needed to actually make a video, not compile one. I create videos so infrequently, that I often forget the name of the video editing software I used last time so it takes some searching. This time I found that I had openshot-qt installed but when I tried to run it, I got a back trace (which someone else has already reported).

I considered looking for another GUI editor, but I wasn’t that interested in learning what might be a complicated user interface when what I need is so simple.

So I kept searching and found melt. Wow.

I ran:

melt originals/* -consumer avformat:all.webm acodec=libopus vcodec=libvpx

And a while later I had a video. Impressive. It handled people who submitted their videos in portrait mode on their cell phones in mp4 as well as web cam submissions using webm/vp9 on landscape mode.

Thank you melt developers!

Welcome to the Lazy Bookkeeper

Thu, 04 Apr 2019 12:18:47 +0000

Welcome to the first (and given my laziness, possibly last) installment of the lazy bookkeeper. I’m actually not really a bookkeeper at all, so if you are looking for a legal and sound way to keep your books, don’t read this post.

I am lazy though.

I’m tasked with keeping the books on a US organization with a budget of about $150,000. I hate doing bookkeeping so I’ve naturally tried to automate everything I possibly can. I’ll explain how I do it in this blog.

The first two (and most important) steps are to use GnuCash and the GnuCash Python bindings. Last I checked about a year ago, the Python bindings were not working with the Python3 version of GnuCash so you may need to hold back on that version.

Income

Income is easier then expenses, perhaps because we have so little of it :).

The only source of income for the organization are:

Membership dues (accounting for almost everything)
Donations
Program related income (e.g. contracts)
Interest (like $10 every year)

We get income via credit card payments (both Stripe and Paypal) and checks.

We have two external systems that handle Membership dues (our custom control panel software) and Donations (CiviCRM).

The custom control panel software automatically generates new membership invoices and sends reminders to people to pay them.

When I get checks, I:

Take a picture of the check with my phone
Copy it to a Nextcloud account in a folder called: Checks to be processed
Enter the checks manually in either our control panel or CiviCRM databases so the sender can be properly credited (these are our single sources of truth for such things)
Deposit the checks and take a picture of the receipt (and share it with the same next Nextcloud folder as the checks)

At this point in the process, I had to struggle against my lazy instincts to take an additional step to solve two problems:

The checks are named based on date/time I took them picture. Not that helpful if I’m looking at a list of them
Entering the checks manually is a source of typos and mistakes - which is a substantial cause of misery when bookkeeping.

So, I’ve written a simple script that reads all the pictures of all the checks and then displays them one by one.

For each check that is displayed, I am prompted to enter the check number. The script then queries the control panel for that check number and displays the member it was coded against and the amount. I check these details with the check itself and confirm it.

Then, the script moves the check from the “checks to be processed” folder to a new folder (in a YYYY/MM format) and renames it to include the check number, short version of the organization it is applied against and the amount in the name itself.

The last step is that every few weeks I run a script to import data from both the control panel and the CiviCRM database.

The script finds control panel invoices and payments and donations in CiviCRM and automatically creates the records in GnuCash.

Expenses

Expenses have been a much harder nut to crack. For many years I simply put all receipts in a giant folder and then, after waiting until the last possible moment, I would go through them all an enter them. I made a Python script to facilitate it - it asks how much the expense is for, what category it should be coded to, etc. However, this task was particularly tedious because I usually couldn’t remember what any expense was for.

So, this year I’ve instituted a new system. I have a set of folders named after each expense account. Now, when I get a receipt, I name it in the format: invoice-number_brief-description_YYYY-MM-DD_00.00_bank.pdf and I save it in the folder matching the expense account.

In short, I am doing the sorting every time a receipt comes in rather than waiting until the end of the year.

Then, I have a script that validates all of my names (in case I made any typos) and automatically imports everything in to GnuCash.

The only other expense to deal with is payroll, so I’ve naturally scripted that as well. I do have to run it for every payroll and enter the variations in random state taxes and such, but by using a terminal script I don’t have to click around the GnuCash interface.

Reconciliation

The only part I can’t automate is the reconciliation. However, by minimizing typos it is much easier. Basically consisting of:

Entering interest and bank fees
Tracking down expenses I didn’t get receipts for

The scripts

These scripts can’t be directly used by anyone else and will be useless if you think of them as general purpose scripts. However, if you want to do something similar for your own purposes, you may find them helpful as a way to see how certain items can be imported into GnuCash.

https://code.mayfirst.org/mfmt/gnucash-import-scripts

I didn't know what ibus was one day ago, now I love it

Mon, 11 Feb 2019 15:50:56 +0000

[See update below.]

After over a decade using mutt as my email client, I finally gave up pretending I didn’t want to see pretty pictures in my email and switched to Thunderbird.

Since I don’t write email in spanish often, I didn’t immediately notice that my old dead key trick for typing spanish accent characters didn’t work in Thunderbird like it does in vim or any terminal program.

I learned many years ago that I could set a special key via my openbox autostart script with xmodmap -e 'keysym Alt_R = Multi_key'. Then, if I wanted to type é, I would press and hold my right alt key while I press the apostrophe key, let go, and press the e key. I could get an ñ using the same trick but press the tilde key instead of the apostrophe key. Pretty easy.

When I tried that trick in Thunderbird I got an upside down e. WTF.

I spent about 30 minutes clawing my way through search results on several occassions over the course of many months before I finally found someone say: “I installed the ibus package, rebooted and it all worked.” (Sorry Internet, I can’t find that page now!)

ibus? apt-cache show ibus states:

IBus is an Intelligent Input Bus. It is a new input framework for the Linux
OS. It provides full featured and user friendly input method user
interface. It also may help developers to develop input method easily.

Well, that’s succinct. I still had no idea what ibus was, but it sounded like it might work. I followed those directions and suddenly in my system tray area, there was a new icon. If I clicked on it, it listed by default my English keyboard.

I could right click, hit preferences and add a new keyboard:

English - English (US, International with dead keys)

Now, when I select that new option, I simply press my right alt key and e (no need for the apostrophe) and I get my é. Same with ñ. Hooray!

My only complaint is that while using this keyboard, I can’t using regular apostrophes or ~’s. Not sure why, but it’s not that hard to switch.

As far as I can tell, ibus tries to abstract some of the difficulties around input methods so it’s easier on GUI developers.

Update 2019-02-11

Thanks, Internet, particularly for the comment from Alex about how I was choosing the wrong International keyboard. Of course, my keyboard does not have dead keys, so I need to choose the one called “English (intl., with AltGr dead keys).”

Now everything works perfectly. No need for ibus at all. I can get é with my right alt key followed by e. It works in my unicode terminal, thunderbird, and everywhere else that I’ve tried.

Identifying resource hogs

Wed, 24 Oct 2018 11:22:33 +0000

Suppose you manage a multi-user server with wily and unpredictable users running compromisable web sites and email accounts with loads of email.

Suppose your server blows up at 3:03 pm.

Suppose you ask: Was there a single compromised user account that caused this mess and if so, which user?

How would you figure it out?

sysstat

If you are lucky enough to be logged into the server when it is blowing up and lucky enough that the server is responsive enough to run commands for you, then you have some options.

vmstat 1 nicely shows you whether you are cpu bound or disk i/o bound. The first two columns show number of processes waiting to run due to either not enough CPU (r) or not enough disk i/o (b).

pidstat -d 1 shows which processes are writing and reading to disk and when you ctrl-c cancel that command it summarizes what you have been watching allowing you to more easily pin point who is doing the most writing.

And good ole top can show you which users are consuming the most CPU.

But, these tools are often misleading. When your system is under heavy contention, all kinds of processes get backed up and these tools often just show a mess of processes desperately trying to run amidst a giant resource shortage. It’s hard to pinpoint the user that may have started the problem.

Also, these tools are useless if you can’t login to the server or if you arrive at the scene after the storm has passed.

munin and sar

Both munin and sar (provided by sysstat) can record a history of usage. And, both can tell you, for example, whether your system was CPU bound or disk i/o bound and exactly when the problem started.

However, I can’t seem to convince either (out of the box at least) how to track such usage on a per user basis.

Gnu Accounting utilities

Now we are getting somewhere. The acct package is specifically designed to record usage information on a per user basis.

However, it suffers from a few problems considering our use case:

It has a subtley different goal: acct wants to account for total resource usage at the end of the day. I want to measure per user resource usage at an exact point and time.

The acctpackage works in a elegant fashion. It enables a feature of the kernel that causes the kernel to add data to a file every time a process ends. The data includes the pid, uid, total cpu usage and average memory consumption and the date and time the process began.

This approach means you don’t have to poll running processes and you always get accurate information.

For the purposes of pin-pointing who is consuming resources when, this works great for short running processes.

But for long running processes, if you chart it by the date/time provided (which is the time the process started), you get a giant jump in resource usage when the process starts. If a process runs for 30 minutes and consumes massive resources during the last minute of it’s life, that resource usage will get reported 29 minutes before it happened.
Confusing to munin. The date/time on the data is when the process started, however, it is reported to the kernel file when the process ends. For the purposes of munin graphing, we would have to record it when the process ends or else we would be reporting times in the past.
A much bigger problem is lack of disk i/o. Although the spec seems to include disk i/o, reporting disk i/o does not seem to be available on linux and sadly disk i/o is almost always the caues of our resource problems. The “io” column in dump-acct is always 0.
The information is reported in binary form making the raw file a bit hard to read. And the tools that come with the acct package interpret that file (thanks!) but do so in a way that is hard to parse (in particular, dates are human readable and only include when the process began, not when it started, so you can’t effecitvely limit output by date range).

Alternatives

I’ve spent years writing various polling programs that either use pidstat or it’s ilk (or directly access pid statistics via /proc) in cron jobs or constantly running processes and collect and record it’s output. However, all these scripts suffer from either being inaccurate because they depend on polling running processes or overly resource consuming themselves because they are in a constant loop measuring things.

Which is faster, rsync or rdiff-backup?

Mon, 13 Aug 2018 12:22:41 +0000

Surprise: rdiff-backup (given our particular constraints).

As our data grows (and some filesystems balloon to over 800GBs, with many small files) we have started seeing our night time backups continue through the morning, causing serious disk i/o problems as our users wake up and regular usage rises.

For years we have implemented a conservative backup policy - each server runs the backup twice: once via rdiff-backup to the onsite server with 10 days of increments kept. A second is an rsync to our offsite backup servers for disaster recovery.

Simple, I thought. I will change the rdiff-backup to the onsite server to use the ultra fast and simple rsync. Then, I’ll use borgbackup to create an incremental backup from the onsite backup server to our off site backup servers. Piece of cake. And with each server only running one backup instead of two, they should complete in record time.

Except, some how the rsync backup to the onsite backup server was taking almost as long as the original rdiff-backup to the onsite server and rsync backup to the offsite server combined. What? I thought nothing was faster than the awesome simplicity of rsync, especially compared to the ancient python-based rdiff-backup, which hasn’t had an upstream release since 2009.

Turns out that rsync is not faster if disk i/o on the target server is your bottle neck.

By default, rsync determines if a file needs to be updated by comparing the time stamp and size of the files on both the source and the target server. That means rsync has to read the meta data on every single file on the source and every single file on the target. At first glance, this would seem faster than rdiff-backup, which compares sha1 checksums (it has to read the entire file, not just the metadata). And, this is definitely the case the first time rdiff-backup runs. However, rdiff-backup has a trick up its sleave: the rdiff-backup-data/mirror_metadata file.

As rdiff-backup runs, it keeps track of the sha1 checksum of every file it backups up in the mirror_medata file on the target. It seems that the next time it runs, it simply compares the sha1 on the source with the sha1 in this file, meaning it doesn’t have to read each file on the target. The result: significantly less disk i/o on the target for faster backups (there is more disk i/o on the source, though, since rdiff-backup has to calculate the sha1 checksum instead of just collecting the size and last modified time stamp).

rdiff-backup also wins by saving all metadata (file ownership and permissions). Since we backup to a non-privileged user on the backup server, this data is lost with rsync. And, for reasons of simplicity, I appreciate having the backup files via a plain filesystem (unlike borgbackup which requires special commands just to get a listing of the files).

For the long term, filesystem-based backup tools seem like a losing proposition compared with block-based backups (like drbd). However, until we can re-organize our data to take advantage of drdb, we will be sticking with rdiff-backup.

Diversity doesn't help the bottom line

Mon, 30 Apr 2018 09:54:09 +0000

A Google software engineer’s sexist screed against diversity has been making the rounds lately.

Most notable are the offensive and mis-guided statements about gender essentialism, which honestly make the thing hard to read at all.

What seems lost in the hype, however, is that his primary point seems quite accurate. In short: If Google successfully diversified it’s workforce, racial and gender tensions would increase not decrease, divisiveness would spread and, with all liklihood, Google could be damaged.

Imagine what would happen if the thousands of existing, mostly male, white and Asian engineers, the majority of whom are convinced that they play no part in racism and sexism, were confronted with thousands of smart and ambitious women, African Americans and Latinos who were becoming their bosses, telling them to work in different ways, and taking “their” promotions.

It would be a revolution! I’d love to see it. Google’s bosses definitely do not.

That’s why none of the diversity programs at Google or any other major tech company are having any impact - because they are not designed to have an impact. They are designed to boost morale and make their existing engineers feel good about what they do.

Google has one goal: to make money. And one strategy: to design software that people want to use. One of their tactics that is highly effective is building tightly knit groups of programmers who work well together. If the creation of hostile, racist and sexist environments is a by-product - well, it’s not one that affects their bottom line.

Would Google make better software with a more diverse group of engineers? Definitely! For one, if African American engineers were working on their facial recognition software, it’s doubtful it would have mistaken people with black faces for gorillas.

However, if the perceived improvement in software outweighed the risks of diversification, then Google would not waste any time on feel-good programs and trainings - they would simply build a jobs pipeline and change their job outreach programs to recruit substantially more female, African Americans and Latino candidates.

In the end, this risk avoidance and failure to perceive the limitations of homogeneity is the achiles heel of corporate software design.

Our challenge is to see what we can build outside the confines of corporate culture that prioritizes profits, production efficiency, and stability. What can we do with teams that are willing to embrace racial and gender tension, risk diviseveness and be willing to see benefits beyond releasing version 1.0?

Procrastinating by tweaking my desktop with devilspie2

Mon, 30 Apr 2018 09:48:05 +0000

Tweaking my desktop seems to be my preferred form of procrastination. So, a blog like this is a sure sign I have too much work on my plate.

I have a laptop. I carry it to work and plug it into a large monitor - where I like to keep all my instant or near-instant communications displayed at all times while I switch between workspaces on my smaller laptop screen as I move from email (workspace one), to shell (workspace two), to web (workspace three), etc.

When I’m not at the office, I only have my laptop screen - which has to accomdate everything.

I soon got tired of dragging things around everytime I plugged or unplugged the monitor and starting accumulating a mess of bash scripts running wmctrl and even calling my own python-wnck script. (At first I couldn’t get wmctrl to pin a window but I lived with it. But when gajim switched to gtk3 and my openbox window decorations disappeared, then I couldn’t even pin my window manually. NOTE: This behavior has changed - I now have my openbox decorations back on my gajim windows.)

Now I have the following simpler setup.

Manage hot plugging of my monitor.

Symlink to my monitor status device:

0 jamie@turkey:~$ ls -l ~/.config/turkey/monitor.status 
lrwxrwxrwx 1 jamie jamie 64 Jan 15 15:26 /home/jamie/.config/turkey/monitor.status -> /sys/devices/pci0000:00/0000:00:02.0/drm/card0/card0-DP-1/status
0 jamie@turkey:~$

Create a udev rule to handle things when the monitor is unplugged (for some reason, automating the detection of the monitor being plugged in was too unreliable).

0 jamie@turkey:~$ cat /etc/udev/rules.d/90-vga.rules 
# When a monitor is plugged in, adjust my display to take advantage of it
ACTION=="change", SUBSYSTEM=="drm", ENV{HOTPLUG}=="1", RUN+="/etc/udev/scripts/vga-adjust"
0 jamie@turkey:~$

And here is the udev script:

0 jamie@turkey:~$ cat /etc/udev/scripts/vga-adjust 
#!/bin/bash

logger -t "jamie-udev" "Monitor event detected, waiting 1 second for \
system to detect change."
sleep 1 

# This process seems to be unreliable when alerting the X11 system, so I'm
# only configuring it to tell the X windows system when the monitor is unplugged.
#
# The X11 system adjusts for a monitor being plugged in via the systemd service
# devilspie2 (~/.config/systemd/user/devilspie2.service) which executes a-vga-auto
# via a PreExec command.
#
# We don't know whether the VGA monitor is being plugged in or unplugged so we
# have to autodetect first. And,it takes a few seconds to assess whether the
# monitor is there or not, so sleep for 1 second.
monitor_status="/home/jamie/.config/turkey/monitor.status"
status=$(cat "$monitor_status")  

export XAUTHORITY=/home/jamie/.Xauthority
export DISPLAY=:0
if [ "$status" = "disconnected" ]; then
  # The monitor is not plugged in	
  logger -t "jamie-udev" "Monitor is being unplugged"
  /usr/bin/xrandr --output DP-1 --off
fi
0 jamie@turkey:~$

Move windows into place.

So far, this handles ensuring the monitor is activated and placed in the right position. But, nothing has changed in my workspace.

Here’s where the devilspie2 configuration comes in:

==> /home/jamie/.config/devilspie2/00-globals.lua <==
-- Collect some global varibles to be used throughout.
name = get_window_name();
app = get_application_name();
instance = get_class_instance_name();

-- See if the monitor is plugged in or not. If monitor is true, it is
-- plugged in, if it is false, it is not plugged in.
monitor = false;
device = "/home/jamie/.config/turkey/monitor.status"
f = io.open(device, "rb")
if f then
  -- Read the contents, remove the trailing line break.
  content = string.gsub(f:read "*all", "\n", "");
  if content == "connected" then
    monitor = true;
  end
end


==> /home/jamie/.config/devilspie2/gajim.lua <==
-- Look for my gajim message window. Pin it if we have the monitor.
if string.find(name, "Gajim: conversations.im") then
  if monitor then
    set_window_geometry(1931,31,590,1025);
    pin_window();
  else
    set_window_workspace(4);
    set_window_geometry(676,31,676,725);
    unpin_window();
  end
end

==> /home/jamie/.config/devilspie2/grunt.lua <==
-- grunt is the window I use to connect via irc. I typically connect to
-- grunt via a terminal called spade, which is opened using a-terminal-yoohoo
-- so that bell actions cause a notification. The window is called spade if I
-- just opened it but usually changes names to grunt after I connect via autossh
-- to grunt. 
--
-- If no monitor, put spade in workspace 2, if monitor, then pin it to all
-- workspaces and maximize it vertically.

if instance == "urxvt" then
  -- When we launch, the terminal is called spade, after we connect it
  -- seems to get changed to jamie@grunt or something like that.
  if name == "spade" or string.find(name, "grunt:") then
    if monitor then
      set_window_geometry(1365,10,570,1025);
      set_window_workspace(3);
      -- maximize_vertically();
      pin_window();
    else
      set_window_geometry(677,10,676,375);
      set_window_workspace(2);
      unpin_window();
    end
  end
end

==> /home/jamie/.config/devilspie2/terminals.lua <==
-- Note - these will typically only work after I start the terminals
-- for the first time because their names seem to change.
if instance == "urxvt" then
  if name == "heart" then
    set_window_geometry(0,10,676,375);
  elseif name == "spade" then
    set_window_geometry(677,10,676,375);
  elseif name == "diamond" then
    set_window_geometry(0,376,676,375);
  elseif name == "clover" then
    set_window_geometry(677,376,676,375);
  end
end

==> /home/jamie/.config/devilspie2/zimbra.lua <==
-- Look for my zimbra firefox window. Shows support queue.
if string.find(name, "Zimbra") then
  if monitor then
    unmaximize();
    set_window_geometry(2520,10,760,1022);
    pin_window();
  else
    set_window_workspace(5);
    set_window_geometry(0,10,676,375);
    -- Zimbra can take up the whole window on this workspace.
    maximize();
    unpin_window();
  end
end

It is started (and restartd) with:

0 jamie@turkey:~$ cat ~/.config/systemd/user/devilspie2.service 
[Unit]
Description=Start devilspie2, program to place windows in the right locations.

[Service]
ExecStart=/usr/bin/devilspie2
ExecStartPre=/home/jamie/bin/a-vga-auto

[Install]
WantedBy=multi-user.target
0 jamie@turkey:~$

Which I have configured via a key combination that I hit everytime I plug in or unplug my monitor.

And lastly, the ExecStartPre:

#!/bin/bash

# If the monitor is plugged in, we have to tell the X window system about it
# so we can see our desktop. If it's unplugged, we have tell the X window system
# to re-adjust. This scripts checks whether the monitor is plugged in and
# whether the x window system realizes it. It auto adjusts the X window setting.
#
# This script is invoked by the devilspie2 systemd service 
# (~/.config/systemd/user/devilspie2.service). The devilspie2 systemd server
# is invoked with W-d as configured in ~/.config/openbox/rc.xml.

monitor_status_path="/home/jamie/.config/turkey/monitor.status"
monitor_status=$(cat "$monitor_status_path")

xrandr_status="disconnected"

if /usr/bin/xrandr | grep "DP-1 connected 1920x1080+1366+0" >/dev/null; then
  xrandr_status="connected"
fi

if [ "$monitor_status" = "disconnected" -a "$xrandr_status" = "connected" ]; then
  /usr/bin/xrandr --output DP-1 --off
elif [ "$monitor_status" = "connected" -a "$xrandr_status" = "disconnected" ]; then
  /usr/bin/xrandr --output DP-1 --right-of eDP-1 --auto
fi

The pain of installing custom ROMs on Android phones

Mon, 30 Apr 2018 09:46:16 +0000

A while back I bought a Nexus 5x. During a three-day ordeal I finally got Omnirom installed - with full disk encryption, root access and some stitched together fake Google Play code that allowed me to run Signal without actually letting Google into my computer.

A short while later, Open Whisper Systems released a version of Signal that uses Web Sockets when Google Play services is not installed (and allows for installation via a web page without the need for the Google Play store). Dang. Should have waited.

Now, post Meltdown/Spectre, I worked up the courage to go through this process again. In the comments of my Omnirom post, I received a few suggestions about not really needing root. Hm - why didn’t I think of that? Who needs root anyway? Combining root with full disk encryption was the real pain point in my previous install, so perhaps I can make things much easier. Also, not needing any of the fake Google software would be a definite plus.

This time around I decided to go with LineageOS since it seems to be the most mainstream of the custom ROMs. I found perfectly reasonable sounding instructions.

My first mistake was skipping the initial steps (since I already had TWRP recovery installed I didn’t think I needed to follow them). I went straight to the step of installing LineageOS (including wiping the Cache, System and Data partitions). Unfortunately, when it came time to flash the ROM, I got the error that the ROM is for bullhead, but the hardware I was using is "" (yes, empty string there).

After some Internet searching I learned that the problem is an out-dated version of TWRP. Great, let’s upgrade TWRP. I went back and started from the beginning of the LineageOS install instructions. But when it came to the fastboot flashing unlock step, I got the message explaining that my phone did not have OEM unlocking enabled. There are plenty of posts in the Internet demonstrating how to boot into your phone and flip the switch to allow OEM unlocking from the Developer section of your System tools. Great, except that I could no longer boot into my phone thanks to the various deletion steps I had already taken. Dang. Did just brick my phone?

I started thinking through how to buy a new phone.

Then, I did more searching on the Internet and learned that I can flash a new version of TWRP the same way you flash anything else. Phew! New TWRP flashed and new LineageOS ROM installed! And, my first question: what is the point of locking your phone if you can still flash recovery images and even new ROMs?

However, when I booted, I got an “OS vendor mismatch error”. WTF. Ok, now my phone is really bricked.

Fortunately, someone not only identified this problem but contributed an exceptionally well-written step-by-step set of directions to fix the problem. The post, in combination with some comments on it, explains that you have to download the Google firmware that corresponds to the error code in your message (in case that post ever goes away: unzip the file you download, then cd into the directory created and unzip the file that starts with image-bullhead. Then, minimally flash the vendor.img to the vendor partition in TWRP).

In other words, the LineageOS ROM depends on having the right Google firmware installed.

All of these steps were possible without unlocking the phone. However, when I tried to update the Radio and Bootloader using:

fastboot flash bootloader bootloader-bullhead-bhz11l.img
fastboot flash radio radio-bullhead-m8994f-2.6.37.2.21.img

It failed, so I booted into my now working install, enabled OEM unlock, unlocked the phone (which wiped everything so I had to start over) and then it all worked.

And, kudos to LineageOS for the simple setup process and ease of getting full disk encryption.

Now that I’m done, I am asking myself a few questions:

I have my own custom ROM and I am not trusting Google with everything anymore. Hooray! So … who am I trusting? This question I know the answer to (I think…):

Team Win, which provides the TWRP recovery software has total control of everything. Geez, I hope these people aren’t assholes.
Google, since I blindly install their firmware vendor image, bootloader image and radio image. I guess they still can control my phone.
Fdroid, I hope they vette their packages, because I blindly install them from their default archives.
Guardian Project, since I enable their fdroid repo too - but hey at least I have met a few of these people and they are May First/People Link members.
Firefox, I download firefox directly from Mozilla since fdroid doesn’t seem to really support them.
Signal, since I download that APK directly as well.
And the https certificate system (which pretty much means Let’s Encrypt nowadays - since nearly everything depends on the integrity of the packages I am downloading over https.

But I’m still not sure about one more question:

Should I lock my phone? Given what I just accomplished without locking it, it seems that locking the phone could make my life harder the next time I upgrade and doesn’t really stop someone else from replacing key components of my operating system without me knowing it.

Certain Days

Wed, 21 Feb 2018 09:23:28 +0000

For those of you thumbing through the last few pages of your 2005 calendar, wondering if your uncle is going to send you another Sierra Club calendar with chipmunks and pine trees, do yourself a favor and get a copy of the Certain Days 2006 Freedom for Policial Prisoners Calendar. It looks better. And it actually supports groups doing political prisoner work.

http://www.certaindays.org

Docker in Debian

Tue, 10 Oct 2017 12:10:04 +0000

It’s not easy getting Docker to work in Debian.

It’s not in stable at all:

0 jamie@turkey:~$ rmadison docker.io
docker.io  | 1.6.2~dfsg1-1~bpo8+1 | jessie-backports | source, amd64, armel, armhf, i386
docker.io  | 1.11.2~ds1-5         | unstable         | source, arm64
docker.io  | 1.11.2~ds1-5         | unstable-debug   | source
docker.io  | 1.11.2~ds1-6         | unstable         | source, armel, armhf, i386, ppc64el
docker.io  | 1.11.2~ds1-6         | unstable-debug   | source
docker.io  | 1.13.1~ds1-2         | unstable         | source, amd64
docker.io  | 1.13.1~ds1-2         | unstable-debug   | source
0 jamie@turkey:~$

And a problem with runc makes it really hard to get it working on Debian unstable.

These are the steps I took to get it running today (2017-10-10).

Remove runc (allow it to remove containerd and docker.io):

sudo apt-get remove runc

Install docker-runc (now in testing)

sudo apt-get install docker-runc

Fix containerd package to depend on docker-runc instead of runc:

mkdir containerd
cd containerd
apt-get download containerd 
ar x containerd_0.2.3+git20170126.85.aa8187d~ds1-2_amd64.deb
tar -xzf control.tar.gz
sed -i s/runc/docker-runc/g control
tar -c md5sums control | gzip -c > control.tar.gz
ar rcs new-containerd.deb debian-binary control.tar.gz data.tar.xz
sudo dpkg -i new-containerd.deb

Fix docker.io package to depend on docker-runc instead of runc.

mkdir docker
cd docker
apt-get download docker.io
ar x docker.io_1.13.1~ds1-2_amd64.deb
tar -xzf control.tar.gz
sed -i s/runc/docker-runc/g control
tar -c {post,pre}{inst,rm} md5sums control | gzip -c > control.tar.gz
ar rcs new-docker.io.deb debian-binary control.tar.gz data.tar.xz
sudo dpkg -i new-docker.io.deb

Symlink docker-runc => runc

sudo ln -s /usr/sbin/docker-runc /usr/sbin/runc

Keep apt-get from upgrading until this bug is fixed:

printf "# Remove when docker.io and containerd depend on docker-runc
# instead of normal runc
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877329
Package: runc 
Pin: release * 
Pin-Priority: -1 

Package: containderd 
Pin: release * 
Pin-Priority: -1 

Package: docker.io
Pin: release * 
Pin-Priority: -1" | sudo tee /etc/apt/preferences.d/docker.pref

Thanks to coderwall for tips on manipulating deb files.

Late to the Raspberry Pi party

Thu, 18 May 2017 23:37:06 +0000

I finally bought my first raspberry pi to setup as a router and wifi access point.

It wasn’t easy.

I first had to figure out what to buy. I think that was the hardest part.

I ended up with:

Raspberry PI 3 Model B A1.2GHz 64-bit quad-core ARMv8 CPU, 1GB RAM (Model number: RASPBERRYPI3-MODB-1GB)
Transcend USB 3.0 SDHC / SDXC / microSDHC / SDXC Card Reader, TS-RDF5K (Black). I only needed this because I don’t have one already and I will need a way to copy a raspbian image from my laptop to a micro SD card.
Centon Electronics Micro SD Card 16 GB (S1-MSDHC4-16G). This is the micro sd card.
Smraza Clear case for Raspberry Pi 3 2 Model B with Power Supply,2pcs Heatsinks and Micro USB with On/Off Switch. And this is the box to put it all in.

I already have a cable matters USB to ethernet device, which will provide the second ethernet connection so this device can actually work as a router.

I studiously followed the directions to download the raspbian image and copy it to my micro sd card. I also touched a file on the boot partition called ssh so ssh would start automatically. Note: I first touched the ssh file on the root partition (sdb2) before realizing it belonged on the boot partition (sdb1). And, despite ambiguous directions found on the Internet, lowercase ‘ssh’ for the filename seems to do the trick.

Then, I found the IP address with the help of NMAP (sudo nmap -sn 192.168.69.*) and tried to ssh in but alas…

Connection reset by 192.168.69.116 port 22

No dice.

So, I re-mounted the sdb2 partition of the micro sd card and looked in var/log/auth.log and found:

May  5 19:23:00 raspberrypi sshd[760]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
May  5 19:23:00 raspberrypi sshd[760]: fatal: No supported key exchange algorithms [preauth]
May  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format
May  5 19:23:07 raspberrypi sshd[762]: error: Could not load host key: /etc/ssh/ssh_host_rsa_key
May  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format
May  5 19:23:07 raspberrypi sshd[762]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key
May  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format
May  5 19:23:07 raspberrypi sshd[762]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
May  5 19:23:07 raspberrypi sshd[762]: error: key_load_public: invalid format

How did that happen? And wait a minute…

0 jamie@turkey:~$ ls -l /mnt/etc/ssh/ssh_host_ecdsa_key
-rw------- 1 root root 0 Apr 10 05:58 /mnt/etc/ssh/ssh_host_ecdsa_key
0 jamie@turkey:~$ date
Fri May  5 15:44:15 EDT 2017
0 jamie@turkey:~$

Are the keys embedded in the image? Isn’t that wrong?

I fixed with:

0 jamie@turkey:mnt$ sudo rm /mnt/etc/ssh/ssh_host_*
0 jamie@turkey:mnt$ sudo ssh-keygen -q -f /mnt/etc/ssh/ssh_host_rsa_key -N '' -t rsa
0 jamie@turkey:mnt$ sudo ssh-keygen -q -f /mnt/etc/ssh/ssh_host_dsa_key -N '' -t dsa
0 jamie@turkey:mnt$ sudo ssh-keygen -q -f /mnt/etc/ssh/ssh_host_ecdsa_key -N '' -t ecdsa
0 jamie@turkey:mnt$ sudo ssh-keygen -q -f /mnt/etc/ssh/ssh_host_ed25519_key -N '' -t ed25519
0 jamie@turkey:mnt$

NOTE: I just did a second installation and this didn’t happen. Maybe something went wrong as I experiment with SSH vs ssh on the boot partition?

Then I could ssh in. I removed the pi user account and added my ssh key to /root/.ssh/authorized_keys and put a new name “mondragon” in the /etc/hostname file.

And… I upgraded to Debian stretch and rebooted.

Then, I followed these instructions for fixing the wifi (replacing the firmware does still work for me).

I plugged my cable matters USB/Ethernet adapter into the device so it would be recognized, but left it dis-connected.

Next I started to configure the device to be a wifi access point using this excellend tutorial, but decided I wanted to setup my networks using systemd-networkd instead.

Since /etc/network/interaces already had eth0 set to manual (because apparently it is controlled by dhcpcd instead), I didn’t need any modifications there.

However, I wanted to use the dhcp client built-in to systemd-networkd, so to prevent dhcpcd from obtaining an IP address, I purged dhcpcd:

apt-get purge dhcpcd5

I was planning to also use systemd-networkd to name the devices (using *.link files) but nothing I could do could convince systemd to rename them, so I gave up and added /etc/udev/rules.d/70-persistent-net.rules:

	SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="b8:27:eb:ce:b5:c3", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME:="wan"
	SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="a0:ce:c8:01:20:7d", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME:="lan"
	SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?*", ATTR{address}=="b8:27:eb:9b:e0:96", ATTR{dev_id}=="0x0", ATTR{type}=="1", NAME:="wlan"

(If you are copying and pasting the mac addresses will have to change.)

Then I added the following files:

root@mondragon:~# head /etc/systemd/network/*
==> /etc/systemd/network/50-lan.network <==
[Match]
Name=lan

[Network]
Address=192.168.69.1/24

==> /etc/systemd/network/55-wlan.network <==
[Match]
Name=wlan

[Network]
Address=10.0.69.1/24

==> /etc/systemd/network/60-wan.network <==
[Match]
Name=wan

[Network]
DHCP=v4
IPForward=yes
IPMasquerade=yes
root@mondragon:~#

Sadly, IPMasquerade doesn’t seem to work either for some reason, so…

root@mondragon:~# cat /etc/systemd/system/masquerade.service 
[Unit]
Description=Start masquerading because Masquerade=yes not working in wan.network.

[Service]
Type=oneshot
ExecStart=/sbin/iptables -t nat -A POSTROUTING -o wan -j MASQUERADE

[Install]
WantedBy=network.target
root@mondragon:~#

And, systemd DHCPServer worked, but then it didn’t and I couldn’t figure out how to debug, so…

apt-get install dnsmasq

Followed by:

root@mondragon:~# cat /etc/dnsmasq.d/mondragon.conf 
# Don't provide DNS services (unbound does that).
port=0

interface=lan
interface=wlan

# Only provide dhcp services since systemd-networkd dhcpserver seems
# flakey.
dhcp-range=set:cable,192.168.69.100,192.168.69.150,255.255.255.0,4h
dhcp-option=tag:cable,option:dns-server,192.168.69.1
dhcp-option=tag:cable,option:router,192.168.69.1

dhcp-range=set:wifi,10.0.69.100,10.0.69.150,255.255.255.0,4h
dhcp-option=tag:wifi,option:dns-server,10.0.69.1
dhcp-option=tag:wifi,option:router,10.0.69.1

root@mondragon:~#

It would probably be simpler to have dnsmasq provide DNS service also, but I happen to like unbound:

apt-get install unbound

And…

root@mondragon:~# cat /etc/unbound/unbound.conf.d/server.conf 
server:
    interface: 127.0.0.1
    interface: 192.168.69.1
    interface: 10.0.69.1

    access-control: 192.168.69.0/24 allow
    access-control: 10.0.69.0/24 allow

    # We do query localhost for our stub zone: loc.cx
    do-not-query-localhost: no

    # Up this level when debugging.
    log-queries: no
    logfile: ""
    #verbosity: 1

    # Settings to work better with systemcd
    do-daemonize: no
    pidfile: ""
root@mondragon:~#

Now on to the wifi access point.

apt-get install hostapd

And the configuration file:

root@mondragon:~# cat /etc/hostapd/hostapd.conf
# This is the name of the WiFi interface we configured above
interface=wlan

# Use the nl80211 driver with the brcmfmac driver
driver=nl80211

# This is the name of the network
ssid=peacock

# Use the 2.4GHz band
hw_mode=g

# Use channel 6
channel=6

# Enable 802.11n
ieee80211n=1

# Enable WMM
wmm_enabled=1

# Enable 40MHz channels with 20ns guard interval
ht_capab=[HT40][SHORT-GI-20][DSSS_CCK-40]

# Accept all MAC addresses
macaddr_acl=0

# Use WPA authentication
auth_algs=1

# Require clients to know the network name
ignore_broadcast_ssid=0

# Use WPA2
wpa=2

# Use a pre-shared key
wpa_key_mgmt=WPA-PSK

# The network passphrase
wpa_passphrase=xxxxxxxxxxxx

# Use AES, instead of TKIP
rsn_pairwise=CCMP
root@mondragon:~#

The hostapd package doesn’t have a systemd start up file so I added one:

root@mondragon:~# cat /etc/systemd/system/hostapd.service 
[Unit]
Description=Hostapd IEEE 802.11 AP, IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator
Wants=network.target
Before=network.target
Before=network.service

[Service]
ExecStart=/usr/sbin/hostapd /etc/hostapd/hostapd.conf

[Install]
WantedBy=multi-user.target
root@mondragon:~#

My last step was to modify /etc/ssh/sshd_config so it only listens on the lan and wlan interfaces (listening on wlan is a bit of a risk, but also useful when mucking with the lan network settings to ensure I don’t get locked out).

Re-thinking Web App Security

Wed, 15 Feb 2017 11:56:30 +0000

An organizer friend interested in activating a rapid response network to counter Trump-era ICE raids on immigrants asked me about any existing simple and easy tools that could send out emergency SMS/text message alerts.

I thought about it and ended up writing my first pouchdb web application to accomplish the task. For the curious, you can see it in action and browse the source code. To use it to send SMS, you have to register for a Twilio account - you can get a free account that has very restricted SMS sending capability or pay for full functionality.

The project is unlike anything I have done before.

I chose pouchdb because it stores all of your contacts in your browser not on a server somewhere in the so-called cloud. (You can also choose to sync to a server, a feature I have not yet implemented.)

The implications of storing your data locally are quite profound.

Classic Web App

Let’s first consider the more common web application: You visit a web site (the same web site that your colleagues visit, or in the case of a massive application like gmail.com, the same web site that everyone in the world visits). Then, you login with your own unique username and password, which grants you access to the portion the database that you are suppose to have access to.

For most use-cases, this model is fairly ideal:

If you have a technically competent host, your data is backed up regularly and the database is available nearly 100% of the time
If you have a politically trust-worthy host, your host will notify you and put up a fight before turning any of your data over to a government agent
If you drop your phone in the toilet you can always login from another computer to access your data
If you save your password in your browser and your laptop is stolen, you can always change your password to prevent the thief from accessing your data
You can easily share your data with others by creating new usernames and passwords

However, there are some downsides:

If your host is not technically competent or polically trust-worthy, you could lose all of your data to a hard drive crash or subpoena
Even if your host is competent, all of your data is one previously undiscovered vulnerability away from being hacked
Even if your host is politically trust-worthy, you cannot always stop a subpoena, particularly given the legal escalations of tools like national security letters

pouchdb no sync

Assuming you are accessing your database on a device with an encrypted disk and you manage your own backups, pouchdb without synchoronizing provides the most privacy and autonomy. You have complete control of your data and you are not dependent on any server operator.

However, the trade-offs are harsh:

Availability: if you lose your device, you would need to restore from backup - which is much more difficult than simply logging in from another device
Collaboration: you simply can’t share this data with anyone else

It seems this model is fairly useless except in very tight corner cases.

pouchdb that synchronizes to a server

With this model, you avoid the trade-offs of the no sync model (hooray!). However, you also lose all of the privacy benefits, and it’s even worse: your data can be compromised either via a server breach or via a compromise of any of the devices you are using. If any of these devices lack encrypted disks, then it’s borderline reckless.

On the other hand, you gain a huge benefit in terms of reliability. If the server goes down, loses your data, fails to backup or is taken offline by a legal order, you can still function perfectly well and can optionally choose to sync to a different host.

Conclusions

Ultimately, we need to better evaluate the trade-offs between privacy and availability for each given use of a database and try to make the best decision.

And… keep working on new models. For example, it seems an ideal middle ground would be to sync in a peer-to-peer fashion with our colleagues (see PeerPouch) or sync to a server under your control in your office.

What's Up with WhatsApp?

Fri, 13 Jan 2017 21:02:48 +0000

Despite my jaded feelings about corporate Internet services in general, I was suprised to learn that WhatsApp’s end-to-end encryption was a lie. In short, it is possible to send an encrypted message to a user that is intercepted and effectively de-crypted without the sender’s knowledge.

However, I was even more surprised to read Open Whisper Systems critique of the original story, claiming that it is not a backdoor because the WhatsApp sender’s client is always notified when a message is de-crypted.

The Open Whisper Systems post acknowledges that the WhatsApp sender can choose to disable these notifications, but claims that is not such a big deal because the WhatsApp server has no way to know which clients have this feature enabled and which do not, so intercepting a message is risky because it could result in the sender realizing it.

However, there is a fairly important piece of information missing, namely: as far as I can tell, the setting to notify users about key changes is disabled by default.

So, using the default installation, your end-to-end encrypted message could be intercepted and decrypted without you or the party you are communicating with knowing it. How is this not a back door? And yes, if the interceptor can’t tell whether or not the sender has these notifications turned on, the interceptor runs the risk of someone knowing they have intercepted the message. Great. That’s better than nothing. Except that there is strong evidence that many powerful governments on this planet routinely risk exposure in their pursuit of compromising our ability to communicate securely. And… not to mention non-governmental (or governmental) adversaries for whom exposure is not a big deal.

Furthermore a critical reason for end-to-end encrption is so that your provider does not have the technical capacity to intercept your communications. That’s simply not true of WhatsApp. It is true of Signal and OMEMO, which requires the active participation of the sender to compromise the communication.

Why in the world would you distribute a client that not only has the ability to surpress such warnings, but has it enabled by default?

Some may argue that users regularly dismiss notifications like “fingerprint has changed” and that this problem is the achilles heal of secure communications. I agree. But… there is still a monumental difference between a user absent-mindedly dismissing an important security warning and never seeing the warning in the first place.

This flaw in WhatsApp is a critical reminder that secure communications doesn’t just depend on a good protocol or technology, but on trust in the people who design and maintain our systems.

End-to-End Encrypted group chats via XMPP

Thu, 05 Jan 2017 17:16:49 +0000

It’s been over a year since my colleagues and I at the Progressive Technology Project abandoned Skype, first for IRC and soon after for XMPP. Thanks to the talented folks maintaining conversations.im it’s been a breeze to get everyone setup with accounts (8 Euros/year is quite worth it) and a group chat going.

However, our group chats have not been using end-to-end encryption… until now. It wasn’t exactly painless, so I’m sharing some tips and tricks.

Use either Conversations for Android (f-droid or Play) or Gajim for Windows or Linux. At the time of this writing, these are the only two applications I know of that support OMEMO, the XMPP extension that supports end-to-end encryption. Chat Secure for iOS, however, is just a release away. We managed to get things working with most of us using both Gajim and Conversations. It would probably have been much easier and smoother if everyone were only using Conversations because OMEMO is built-in to core, rather than Gajim, where OMEMO support is provided via an extension.
If you are using Gajim… After installing the OMEMO plugin in Gajim, fully restart Gajim. Similarly, if you add or remove a contact from your group, it seems you have to fully restart Gajim. Not sure why. If something is not working in Gajim, try restarting it.
Ensure that everyone in your group has added everyone else in the group to their roster. This was the single biggest and most confusing part of the process. If you are missing just one contact in your roster, then messages you type into the group chat will not show up without any indication as to what happened or why (on Gajim). Take this step first or prepare for confusing failures. Remember: everyone has to have everyone else in their roster.
Create the group in the android Conversations app, not in Gajim. There are strict requirements for how the group needs to be setup (private, members only and non-anonymous). I tried creating the group in Gajim and followed the directions but couldn’t get it to work. Creating the group in Conversations worked right away. Remember: don’t add members to the group unless everyone has them in their roster!
You can give your group a easy to remember name in your Gajim bookmarks, but under the hood, it will be assigned a random name. Conversations will show you the random name via “Conference Details” and Gajim will show it under the tab in the Messages window. When inviting people to the group you may need to select the random name.
Trust on First Use. In our experiment, we created a group for four people and we were all on a video and voice chat while we set things up. Three out of the four of us had both Gajim and Conversations in play. That meant 4 different people had to verify between 5 and 6 fingerprints each. We decided to use Trust on First Use rather than go through the process of reading out all the fingerprints (for the record, it still took us an hour and 15 minutes to get it all working). See Daniel Gultsch’s interesting article on Trust on First Use.
If you get an error “This is not a group chat” it may be because you accidentally added the group as a contact to your roster. Click View -> Offline contacts. And if you see your group listed, delete it and close the tab in your Messages window (if one is open for it). You may also need to restart Gajim. Repeat until it no longer shows up in your roster.

Anyone interested in secure XMPP may also find the Riseup XMPP page useful.

Should we be pushing OpenPGP?

Thu, 15 Dec 2016 08:53:16 +0000

Bjarni Rúnar, the author of Mailpile released a blog about recent blogs disparaging OpenPGP. It’s a good read.

There’s one reason to support OpenPGP missing from the blog: OpenPGP protects you if your mail server is hacked. I’m sure that Debbie Wasserman Schultz wishes she had been using OpenPGP.

Having said all of this… OpenPGP didn’t make my recent list of security tips. That ommission is for two reasons:

I’ve never trusted my phone enough to store my OpenPGP keys on it. However, now that I am encrypting my data partition on the phone, should I re-consider? I use the K-9 email client which has had OpenPGP support for years, should I recommend that other people use K-9 and upload their keys to their phones? Suggesting that people use OpenPGP without the ability to use it on your phone seems like an empty suggestion. What about OpenPGP on the iPhone?
I’m waiting for Mailiple 1.0 to be released so I have a viable suggestion for how people can start using encryption now on their desktops. The complexity of using Thunderbird with Enigmail (and the uncertain future of Thunderbird) make it a hard sell. Should I re-consider? What about Mailvelope? Should I be encouraging people to use Mailvelope with their Gmail, etc. accounts?

Wait... is that how you are supposed to configure your SSD card?

Thu, 08 Sep 2016 13:43:46 +0000

I bought a laptop with only SSD drives a while ago and based on a limited amount of reading, added the “discard” option to my /etc/fstab file for all partitions and happily went on my way expecting to avoid the performance degradation problems that happen on SSD cards without this setting.

Yesterday, after a several month ordeal, I finally installed SSD drives in one of May First/People Link’s servers and started doing more research to find the best way to set things up.

I was quite surprised to learn that my change in /etc/fstab accomplished nothing. Well, not entirely true, my /boot partition was still getting empty sectors reported to the SSD card.

Since my filesystem is on top of LVM and LVM is on top of an encrypted disk, those messages from the files system to the disk were not getting through. I learned that when I tried to run the fstrim command on one of the partitions and received the message that the disk didn’t support it. Since my /boot partition is not in LVM or encrypted, it worked on /boot.

I then made the necessary changes to /etc/lvm/lvm.conf and /etc/crypttab, restarted and… same result. Then I ran update-initramfs -u and rebooted and now fstrim works. I decided to remove the discard option from /etc/fstab and will set a cron job to run fstrim periodically.

Also, I learned of some security implications of using trim on an encrypted disk which don’t seem to outweigh the benefits.

Trusted Mobile Device: How hard could it be?

Wed, 31 Aug 2016 23:39:41 +0000

I bought a new phone. After my experiences with signal and the helpful comments readers gave regarding the ability to run android and signal without Google Play using microg I thought I would give it a shot.

Since microg reports that signature spoofing is required and comes out-of-the-box with omnirom I thought I’d aim for installing omnirom’s version of Android 6 (marshmallow) after years of using cyanomgenmod’s version of Android.

The Nexus line of phones seemed well-supported by omnirom in particular (and the alternative ROM community in general) so I bought a Nexus 5x.

I carefully followed the directions for installing omnirom however when it came time to boot into omnirom, I just got the boot sequence animation over and over again.

Frustrated, I decided to go back to cyanogenmod and see if I could use one of the microg recommended methods for getting signature spoofing to work. The easiest seemed to be Needle by moosd but alas no such luck with Marshmallow. Someone else forked the code and might fix it one day. I then spent too much time trying to understand what xposed is before I gave up understanding it and just tried to install it (woops, looks like the installer page is out of date so instead I followed sketchy instructions from a forum thread). Well, to make a long story short it resulted in a boot loop.

So, I decided to return to omnirom. After reading some vague references to omnirom and supersu, I decided to flash both of them together and voila, it worked!

Next, I decided to enable full disk encryption. Not so fast. After clicking through the screens and hitting the final confirmation, my phone rebooted and spent the next 5 hours showing me the omnirom boot animation. Somehow, powering down and starting again resulted in a working machine, but no disk encryption.

After much web searching, guessing and trial and error, I fixed the problem by clicking on the SuperSU option to “Full unroot” the device (I pressed “no” when prompted to attempt to restore stock image). Then I rebooted and followed the directions to encrypt the device. And it worked! Hooray!

I had to reboot and re-flash the supersu to regain su privileges.

All was great.

The first root action I decided to take was to install the cryptfs program from f-droid because using the same password to decrypt your device as you use to unlock the screen seems either tedious or insecure.

That process didn’t work so well. I got a message saying: use this command from a root shell before you reboot: vdc cryptfs changepw <password>. I followed the advice, carefully typing in my 12 character password which includes numbers and letters.

Then, I happily did what I expected to be my last reboot when, to my horror, I was prompted to decrypt my disk with … a numeric-only keypad.

That wasn’t going to work. At this point I had already spent 5 days and about 8 hours on this project. Sigh. So, I started over.

Guess what? It only took me 25 minutes but, it seems that cryptfs is broken. Even with a numeric password it fails. Ok, I guess I need a long pin to unlock my phone. This time it only took my 15 minutes to wipe and re-install everything.

There are only two positive things I can think of:

TWRP, which provides the recovery image, is really great. Everytime something went wrong I booted into the TWRP recovery image and could fix anything.
I’m starting to get used to the error on startup warning me that “Your device is corrupt. It can’t be trusted and may not work properly.” It’s a good thing to remember about all digital devices.

p.s. I haven’t even tried to install microg yet… which was the whole point.

Networking in 2016

Sun, 21 Aug 2016 10:20:03 +0000

So many options, so little time.

Many years ago I handled all my network connections via /etc/network/interfaces.

Then, I was in desperate need of a Internet connection and all I had was a borrowed USB cell phone modem. My friends running NetworkManager just plugged the stick in and were online. I was left with the task of figuring out how to manually configure this piece of hardware without being online. I ended up borrowing my friend’s computer. Then, when I got home, I installed NetworkManager.

Once I had NetworkManager installed, I decided it was easier to find, connect to and manage passwords of wireless networks using a graphical tool rather than digging through the copious output of commands run from my terminal and trying to keep track of the passwords. So long wireless.

Then I had to help a colleague get on our Virtual Private Network. Wow. There’s a NetworkManager GUI for that too. If I’m going to support my colleauge with this tool… I guess I should use it as well. I also managed to write a dispatcher script in /etc/NetworkManager/dispatcher.d that calls su -c "/usr/bin/smbnetfs /media/smbnetfs" -l jamie when it receives and action of “vpn-up” and umount /media/smbnetfs 2>/dev/null on “vpn-down.” Now I can mount the samba share by simply connecting to the VPN via NetworkManager.

My cable connections are almost always configured using DHCP. Almost everything else is in NetworkManager, why not move enp1s0f2 as well?

What’s left? My final piece is my bridge. I still run and manage my own KVM guests and I have a bridge to handle that traffic. I first decided to move this functionality to systemd.network because systemd can not only handle the bridge, but can also handle IP Forwarding, DHCP service, and best of all, IP Masquerading. Well, almost… not IP Masquerading after all, at least not yet.

Without IP masquerading, I figured I’d go with NetworkManager. Having all networking in the same place gives me an illusion of control at best and at worst makes it easier to debug. So, I setup a crufty script in /etc/NetworkManager/dispatcher.d that configures masquerading via iptables everytime either my wireless or wired network goes up or down, which I’m not crazy about. Maybe when #787480 is fixed I’ll got back to systemd. I also edited /etc/sysctl.conf to enable #net.ipv4.ip_forward=1. Then I changed it back and added my own file in /etc/sysctl.d to do the same thing. Then I deleted that file and added sysctl net.ipv4.ip_forward=1 and sysctl net.ipv4.ip_forward=0 to my crufty dispatcher script. I decided to do without DHCP - I can manually configure the few KVM instances that I run.

Now /etc/network/interfaces is so lonely.

Update 2017-08-17

For some reason placing su -c "/usr/bin/smbnetfs /media/smbnetfs" -l jamie in my /etc/NetworkManager/dispatcher.d/ script stopped working. According to journalctl:

Aug 21 10:01:13 turkey systemd[1]: session-c18.scope: Killing process 27385 (smbnetfs) with signal SIGTERM.
Aug 21 10:01:13 turkey systemd[1]: session-c18.scope: Killing process 27390 (smbnetfs) with signal SIGTERM.
Aug 21 10:01:13 turkey systemd[1]: session-c18.scope: Killing process 27391 (smbnetfs) with signal SIGTERM.

I have no idea why and wasn’t really sure how to debug it (running the command su -c "/usr/bin/smbnetfs /media/smbnetfs" -l jamie via a root terminal was successful).

So I decided to create a system service:

0 jamie@turkey:~$ cat /etc/systemd/system/mount-borges.service
[Unit]
Description=Mount PTP's SMB file server borges

[Service]
User=jamie
ExecStart=/usr/bin/smbnetfs /media/smbnetfs -f
ExecStop=/bin/umount /media/smbnetfs
0 jamie@turkey:~$

And now instead of calling su -c "/usr/bin/smbnetfs /media/smbnetfs" -l jamie on vpn-up and umount /media/smbnetfs 2>/dev/null on vpn-down I call systemctl start mount-borges and systemctl stop mount-borges instead.

Nice Work Apertium

Wed, 17 Aug 2016 10:07:36 +0000

For the last few years I have been periodically testing out apertium and today I did again and was pleasantly surprised with the quality of the english-spanish and spanish-english translations (and also their nifty web site translator).

So, I dusted off some of my geeky code to make it easier to use and continue testing.

For starters…

	sudo apt-get install apertium-en-es xclip coreutils

Then, I added the following to my .muttrc file:

	macro pager <F2> "<enter-command>set pipe_decode<enter><pipe-entry> sed '1,/^$/d' | apertium es-en | less<enter><enter-command>unset pipe_decode<enter>" "translate from spanish"

If you press F2 while reading a message in spanish it will print out the English translation.

If you use vim, you can create ~/.vim/plugins/apertium.vim with:

	function s:Translate()
		silent !clear
		execute "! apertium en-es " . bufname("%") . " | tee >(xclip)"
	endfunction
	command Translate :call <SID>Translate()

Then, you can type the command:

:Translate

And it will display the English to Spanish translation of the file you are editing and copy the translation into your clip board so you can paste it into your document.

Noam use Gnome

Sat, 13 Aug 2016 21:04:54 +0000

I don’t quite remember when I read John Goerzen’s post about teaching a 4 year old to use the linux command line with audio on planet Debian. According to the byline it was published nearly 2 years before Noam was born, but I seem to remember reading it in the weeks after his birth when I was both thrilled at the prospect of teaching my kid to use the command line and, in my sleepless stupor, not entirely convinced he would ever be old enough.

Well, the time came this morning. He found an old USB key board and discovered that a green light came on when he plugged it in. He was happily hitting the keys when Meredith suggested we turn on the monitor and open a program so he could see the letters appear on the screen and try to spell his name.

After 10 minutes in Libre Office I remembered John’s blog and was inspired to start writing a bash script in my head (I would have to stop the fun with Libre Office to write it so the pressure was on…). In the end it was only a few minutes and I came up with:

#!/bin/bash

while [ 1 ]; do
  read -p "What shall I say? "
  espeak "$REPLY"
done

It was a hit. He said what he wanted to hear and hit the keys, my job was to spell for him.

Oh, also: he discovered key combinations that did things that were unsurprising to me (like taking the screen grab above) and also things that I’m still scratching my head about (like causing a prompt on the screen that said: “Downloading shockwave plugin.” No thanks. And, how did he do that?

Monitoring Deflect

Tue, 19 Jul 2016 13:26:13 +0000

May First/People Link has several members that are targets of politically motivated denial of service attacks (mostly groups that support reproductive justice for women and palestinian rights). To fight off the attacks, we work closely with Deflect - a non-governmental organization based in Canada that fights against this kind of censorship.

When a site is down, it’s not always easy to understand why. Deflect runs as many as 5 edge servers, any of them could be down. And, of course, the origin server could also be down.

I tried using a commericial/free as in beer service for monitoring up time, but when it reported the site being down, I had no idea which part was down.

Here are few of my tips.

First, you usually want to lookup the IP addresses reported by DNS to you can get a list of your edge servers:

dig +short mayfirst.org

Then, you can use httping to test each one, e.g.:

 httping -l --divert-connect 31.24.34.89 https://mayfirst.org/

The -l says to use https (seems redundant if you specify https as the target, but if you don’t put -l, httping will complain).

The –divert-connect option is particularly useful with your origin IP - because you can compare response times and ensure that your origin is up.

This is all fine and good for generally testing your edges, but sometimes it’s a particular page that is causing the problem. We’ve found that edges sometimes cache error pages, even after the error has gone away. But… how can you prove it?

curl --resolve mayfirst.org:443:31.24.34.89 --head https://mayfirst.org/en/contact/

The –resolve argument is the key here - it allows you to specify how a domain should be resolved (with this option, curl does not perform a DNS lookup). Replace the IP address with the origin IP address and you have a nice comparison between the edge and the origin. The caching server should respond with a Via header to confirms it is caching. And, of course, the first line shows you the http response which should be the same on the origin and caching server.

I’ve written the script below to help debug sites that may not be working properly. It combines both the httping and curl options.

a-reverse-proxy-test domain.org [origin-ip]

Here’s the code:

#!/bin/bash

# Test all given edges using httping and curl.  Pass the full protocol, domain
# and page you want to test as the first argument. Optionally pass the origin
# IP address as the second argument (so you can tell if your origin is down).
# For example:
# 
# a-reverse-proxy-test https:/domain.org/contact 1.2.3.4
#

url="$1"
origin="$2"

if [ -z "$url" ]; then
  printf "Please pass the url as first argument.\n"
  exit 1
fi
proto=$(echo "$url" | sed -E "s#(https?).*#\1#")


host=$(echo "$url" | sed -E "s#https?://([^/]+)/?.*#\1#")

if [ -z "$proto" ]; then
  printf "Please include http or https as part of the url.\n"
  exit 1
fi

if [ -z "$host" ]; then
  printf "Failed to extract the host. If passing just host, ensure it ends in a /, e.g. https://mayfirst.org/.\n"
  exit 1
fi

if ! ping -c 1 4.2.2.1 >/dev/null; then
  # printf "We are offline. Not running.\n"
  exit 1
fi

ips=$(dig +short "$host")
if [ "$?" -ne "0" ]; then
  # printf "DNS lookup failure. Not running.\n"
  exit 1
fi
if [ -n "$origin" ]; then
  ips="$ips $origin"
fi

l=
port=80
if [ "$proto" = "https" ]; then
  l=-l
  port=443
fi

for ip in $ips; do
  printf "Testing IP: %s\n" "$ip"
  printf "httping response: %s\n" "$out"
  httping $l -m -t 5 -c 1 --divert-connect "$ip" "$url"
  printf "curl HTTP and Via headers:\n"
  curl --resolve "$host:$port:$ip" --head "$url" 2>&1 | egrep "^(HTTP|Via)"
  printf "\n"
done

To help with regular monitoring (using httping) I wrote a script that runs via a cron job and outputs the results to a log file.

	#!/bin/bash

	# Test all given edges 

	domain="$1"
	origin="$2"
	proto=http
	if [ -n "$3" ]; then
		proto="$3"
	fi

	if [ -z "$domain" ]; then
		printf "Please pass the domain as first argument.\n"
		exit 1
	fi

	if ! ping -c 1 4.2.2.1 >/dev/null; then
		# printf "We are offline. Not running.\n"
		exit 1
	fi

	ips=$(dig +short "$domain")
	if [ "$?" -ne "0" ]; then
		# printf "DNS lookup failure. Not running.\n"
		exit 1
	fi
	if [ -n "$origin" ]; then
		ips="$ips $origin"
	fi

	l=
	if [ "$proto" = "https" ]; then
		l=-l
	fi

	for ip in $ips; do
		date=$(date +%Y.%m.%d-%H:%M)
		for i in 1 2 3; do
			out=$(httping $l -m -t 5 -c 1 --divert-connect "$ip" "$proto://$domain")
			[ -z "$out" ] && out=1
			printf "%s %s %s\n" "$date" "$ip" "$out"
		done
	done

Signal and Mobile XMPP Update

Sat, 04 Jun 2016 23:03:27 +0000

First, many thanks to Planet Debian readers for your thoughtful and constructive feedback to my Signal and Mobile Instant Messaging blogs. I learned a lot.

Particularly useful was the comment directing me to Daniel Gultsch’s The State of Mobile in 2016 post.

I had previously listed the outstanding technical challenges as:

Implement end-to-end encryption
Receive messages the moment they are sent without draining the battery

I am now fairly convined that both problems are well-solved on Android via the Conversations app and a well-tuned XMPP server (I had no idea how easy it was to install your own Prosody modulues – the client state indicator module is only 22 lines of lua code!)

I think the current technical challenges could be better summarized as: adding iOS (iPhone) support. Both end-to-end encryption and receiving messages consistently seem to be hurdles. However, it seems that Chris Ballinger and the Chat Secure team are well on their way toward solving the push issue and facing funder skittishness on the encryption front. Nonetheless, but seem to be progressing.

With the obvious technical hurdles in progress, we have the luxury of talking about the less obvious ones - particularly the ones requiring trade-offs.

In particular: Signal replaces your SMS client. It looks and feels like an SMS client and automatically sends un-encrypted messages to everyone your address book that is not on signal and sends encrypted messages to those that are on signal.

The significance of this feature is hard to over-state. It differentiates tools by and for technically minded people and those designed for a mass audience.

When I convince people to use Conversations, in contrast, I have to teach them to:

Create an entirely new address book by entering addresses for your friends that you don’t already have
Use a new and different app for sending encrypted messages

For most people who don’t (yet) have their friends XMPP addresses or for people who don’t have any friends who use XMPP, it means that they will install it, send me a few messages and then never use it again.

The price Signal pays for this convenience is steep: Signal seems to synchronize your entire address book to their servers so they can keep a map of cell phone numbers to signal users. It’s not only creepy (I get a text message everytime someone in my address book joins Signal) but it’s flies in the face of expectations for a privacy-minded application.

How could we take advantage of this feature, without the privacy problems?

What if…

Our app could send both XMPP messages and SMS messages
Everytime you added a new XMPP contact, it added the contact to your address book with a new XMPP field
Anytime you send a message to a contact with an XMPP field filled in, it would send via XMPP and otherwise it would send a normal SMS message

The main downside (which Signal faces as well) is that you have to contend with the complexities of sending SMS messages on top of the work needed to write a well-functioning XMPP client. As I mentioned in my Signal blog, there are no shortage of MMS bugs against Signal. Nobody wants that head-ache.

Additinally, we would still lose one Signal feature: with Signal, when a user joins, everyone automatically sends them encrypted messages. With this proposed app, each user would have to manually add the XMPP address and have no way of knowing when one of their friends gets an XMPP address.

Any other ideas?

Mobile Instant Messaging

Wed, 01 Jun 2016 21:04:22 +0000

Now that I’ve gone down the signal road, when can I get off of it?

The two contenders I’ve found for more politically conscious mobile-friendly instant messaging are Tox and XMPP.

And the problems to solve are:

Implement end-to-end encryption
Receive messages the moment they are sent without draining the battery

The Tox Project has a great design (peer-to-peer) and handles end-to-end encryption as part of its core design.

However, the Antox mobile client has not only failed to solve the battery drain problem but seems to also have a serious bandwidth issue as well. Also, what is up with this drama?

How about XMPP?

For years, XMPP clients have been using OTR - and many instant messaging applications support it, including ChatSecure. And now, there seems to be quite a bit of excitement around implementing a better protocol called OMEMO. Incidentally, the OMEMO protocol uses the same Double Ratchet (previously referred to as the Axolotl Ratchet) protocol that Signal uses.

While there are some irritating hiccups around using the available GPL library for Double Ratchet on iPhone apps, it seems like these issues are being sorted out and soon we’ll have some kind of standard way for XMPP clients to exchange end-to-end encrypted messages.

Let’s move on to notifications and battery drain.

If you believe the author of the Android Conversations App there doesn’t seem to be a problem. However, client state indication seems to be the only issue related to battery drainage he addresses and I couldn’t figure out how to easily install that extension on our Debian Jessie prosody instance since it isn’t included in the prosody-modules package in Debian and it requires additional modules for it to work.

Chris Ballinger, the author of ChatSecure (iOS and Android XMPP app), is less optimistic about this problem in general. In short, he thinks we need a proper “push” mechanism and has even started to implement one. At the same time, a new (and different) XMPP standard called Push Notifications - XEP-0357 has been released and it is even implemented in Prosody (although not yet available in Debian).

So the future seems bright, right? Well, not exactly. All of this “push” activity seems to solve this problem: A federated/decentralized application cannot properly use Apple’s APNs or Google’s GCM. In the words of Chris Ballinger:

The biggest problem is that there is no easy way to send push messages between my app and your app. For me to send a push to one of your app’s users on iOS, I must first obtain an APNs SSL certificate/key pair from you, and one of your user’s ‘push token’ that uniquely identifies their device to Apple. These push tokens are potentially sensitive information because they allow Apple to locate your device (in order to send it a push).

So, even if we manage to get one of these two standards for push notifications up and running, we have only succeeded in solving Signal’s centralization problem, not the dependence on Google Play Services and Apple Push Network (in fact it’s quite mysterious to me how you could even use the Prosody implementation of Push Notifications with GCM or APN).

So… what we really would need would be to figure out how to implement one of these two push standards and then get it to work with an alternative to GCM and APN (perhaps MQTT)? Which, I think would require changes to the XMPP client.

Geez. I may be on Signal longer than I planned.

Signal and Google Cloud Services

Wed, 01 Jun 2016 19:08:13 +0000

I just installed Signal on my Android phone.

It wasn’t an easy decision. I have been running Cyanogenmod, a Google-free version of Android, and installing apps from F-Droid, a repository of free software android apps, for several years now. This setup allows me to run all the applications I need without Google accessing any of my cell phone data. It has been a remarkably successful experiment leaving me with all the phone software I need. And it’s consistent with my belief that Google’s size, reach and goals are a menace to the left’s ability to develop the autonomous communications systems on the Internet that we need to achieve any meaningful political change.

However, if I want to install Signal, I have to install Google Play, and the only way to install Google Play is to install Google’s base line set of apps and to connect your cell phone to a Gmail account.

Why in the world would Signal require Google Play? There is plenty of discussion of the technical debate on this topic, but politically it boils down to this: security is about trade-offs, and the trade-offs you find important are based on your politics. While I consider Signal to be on the same team in the big picture, I think Signal’s winning a short term victory for more massively adopted end-to-end encryption at the expense of a longer term and more important struggle for autonomous communication systems specifically for communities fighting corporate power (of which Google itself is an important target) and fighting US hegemony on a global scale.

Furthermore, Signal’s lead developer’s outright hostility to alternate clients connecting to the centralized signal servers demonstrates another political decision that favors central control over any confidence in a broader movement control over what could have become a power new protocol for secure communication. And his refusal to grant an exemption for other developers to use just the encryption algorythm is frustrating to say the least.

Given this reasoning, why install Signal? The main reason is because I have yet to convince anyone to remove Google Apps from their phone and Signal, right now, represents a dramatic improvement over most people’s current communications habit. And, when it comes down to it, I need to run what I recommend.

I’m still running both conversations and Antox which are far better alternatives in the long run. However until they gain more widespread adoption, I’ll be experimenting with Signal.

Technical Details

Oh, and by the way, installing Google Apps nearly bricked my phone.

Cyanogenmod’s web site conveniently provides instructions for installing Google Apps. However, confusingly, they provide two different links to choose from, one is from OpenGapps which provided my a link to download the zip file to flash on a non-https page (the link itself was over https). The other link was to an https-enabled page on androidfilehost.com that offered a non-https download (but did provide a md5 checksum). I am now sure why people offering software downloads don’t enable https from start to finish (well, maybe I do - I haven’t yet enabled https on this site…).

However, more confusing is that both links were to different files. The OpenGapps one seemed to be a daily build and the androidfilehost was to a file with the date 20140606 in it’s name, suggesting it was built nearly 2 years ago.

I went with the daily build.

When I restarted, I got the error “Unfortunately, Android Keyboard (Aosp) Has Stopped.” If you search the web for this error you will see loads of people getting it. However, none of them seem to be using an encrypted disk. Yes, that is a bigger problem since you can’t enter your encrypted disk passphrase if your keyboard app has crashed and you can’t boot your phone if you can’t even hit enter at the passphrase prompt. If you can’t boot, you can’t clear the keyboard app cache or most of the suggestions. In fact, when you press and hold the power key you don’t even get the option to reboot into recovery mode. And, if you connect your device to your USB cable and run the adb tool on your computer, the tool reports that you are not authorized to connect your device.

Oh damn. Did I just brick my phone?

Fortunately, you can still boot into recovery mode on a Samsung S4 by powering it off. Then, press and hold the up volume button while turning it on.

In recovery mode, I as able to convince the adb tool to connect to my device and I copied over the other Gapps zip file from androidfilehost.com and flashing that one seems to have fixed the problem.

Once I booted, I ran Google Play and opted to create a new Google Account. I chose the option to not sync my data. Then, I checked in Settings -> Accounts I saw that a Google Account was there and was synchronizing. Great. What was it synchronizing? I clicked the account, then clicked “Accounts and Privacy” and ensured that everything was turned off. Let’s hope that works.

[Update]

Signal’s option to take over as your default SMS client and send un-encrypted normal SMS messages while sending encrypted messages to other Signal users is a very good way to smooth adoption. Unfortunately I had some problems with MMS message for which I found a work-around. But sheesh, lots of MMS problems at the moment.

Docker: evil spawn or useful tool

Sat, 30 Jan 2016 23:44:07 +0000

There are plenty of criticisms of docker, the system for building a container-based virtual machine running just a single application. I’ve read many of them have have consistently been either in agreement or at least amused.

The most relevant criticism is about the basic approach of building single-application virtual machines. To understand this criticism, let’s remember - in traditional application deployment there are at least three distinct jobs:

Distrubtion developers - the team the integrates all the various packages available into a coherent and well-functioning system and, importantly, monitors the upstream of all the packages to ensure that security bugs are properly packaged and made easily available for installation. These people don’t give a shit about your PHP parse error and don’t really care if you can’t figure out the command to create a new Postgres Database.
System administrators - these are the people that ensure that the security updates provided by the distribution developers get installed on a regular basis and if something breaks during this process, they are the ones that fix it. They know how to create your new postgres database. They don’t really care about your PHP parse error either, but may get roped in to tell you it’s a PHP parse error.
Application developers - these are the people that care about the PHP parse error. They also know how to create beautiful things that end users can interact with.

These three groups have lived in happy tension for years.

Now we have containers.

The problem with containers is that suddenly the system administrators are out entirely and the distribution developers’ role may have been dramatically minimized or circumvented altogether. Meanwhile, the application developer is both liberated from any constraints imposed by either the system administrator or the distribution developers (hooray!) but is also saddled with the enormous responsibilities of these two roles - which may not be apparent to the application developer at first.

I have practically no distribution developer experience and have considerable experience as a system administrator and an application developer. And, it took me months to sort through how to properly develop and deploy an application using docker in a way that I thought was responsible and secure.

I started by learning about how docker wants you to deploy images - by downloading them from their shared registry. As I mentioned, I have very little experience in the realm of distribution development, but I at least know enough about Debian to know that a lot of time and thought has gone into cryptographically verifying packages that I install, which apparently is not done at all with Docker images. Is that obvious to everyone using Docker??

Fortunately, you can work around this problem by creating your own base image which is trivially easy. So, now I build all images, from scratch, locally. That helps put Debian developers back into the mix.

Next, I started looking at Docker Files that I could use to construct my images and discovered something else troubling. Take, for example, the official nginx Docker image. It is based on Debian Jessie (hooray - our distribution developers are in the mix!). However, it then proceeds to install nginx from the nginx repository, not the debian repository. Well, I guess if you are nginx you want to have full control, but still, the Debian developers version of nginx has been vetted to ensure it works with Debian Jessie, so you are really losing something here.

So… my next break from Docker convention is to use all Debian packages in my Docker files.

Once my images were built and my application was tested and running, I was done, right?

Wrong. Well, I would have been done if I didn’t care about upgrading my systems, backing up my data or running cron jobs. Remember those things? Those are things that distribution developers and system administrators have been perfecting for decades. And, they are not trivial.

I then built a series of scripts to help alert me when a image I am using has updates (you can’t just use cron-apt any more since cron isn’t running in your container) and help me update the image and deploy it to all my applications (which involves restarting the application). Backing up data is a whole different can of worms - sometimes involving interacting with your container (if it’s a database container, you have to launch a database dump) or simply copying files from the host, assuming you got the right Docker volume strategy (which took me days to fully understand). Lastly, I had to run a cron job from the host that then runs whatever commands are needed on each of my containers.

This was complicated.

In the end, was it worth it? Yes, I think so. However, not because it was simple, which seems to be the Docker mantra. I think it’s worth it because I can run 100 instances of my application using significantly less resources than when I was using full virtualization and because I can more easily and flexibly adjust the resource allocation. However, check in with me in a year and I’ll probably have a different opinion.

Agents, the kinds that work for us.

Mon, 03 Aug 2015 10:37:36 +0000

I recently decided I wanted to adjust how long my gpg password cache lasts and realized I had no idea how either my ssh-agent or gpg-agent was being launched or even which programs were being used as the agent for which protocol. While I appreciate that everthing just works based on blind faith, I decided to investigate more closely.

I started by reading the man pages for ssh-agent and gpg-agent and discovered that gpg-agent could be used for providing ssh-agent.

Since I run openbox, I decided to dig through my ~/.config/openbox/autostart file and discovered that I was also launching gnome-keyring-daemon, which provides both an ssh-agent and a gpg-agent. This is getting confusing.

I then examined the output of ps -eFH and found this:

jamie    28499 28489  0  1082  1516   1 10:06 tty1     00:00:00     /bin/sh /usr/bin/startx
jamie    28542 28499  0  3987  1912   1 10:06 tty1     00:00:00       xinit /etc/X11/xinit/xinitrc -- /etc/X11/xinit/xserverrc :0 vt1 -auth /tmp/serverauth.gn3C5pYbPc
root     28543 28542  0 58065 25036   3 10:06 tty1     00:00:05         /usr/bin/X -nolisten tcp :0 vt1 -auth /tmp/serverauth.gn3C5pYbPc
jamie    28548 28542  0 69563 19980   0 10:06 tty1     00:00:00         /usr/bin/openbox --startup /usr/lib/x86_64-linux-gnu/openbox-autostart OPENBOX
jamie    28604 28548  0  2774   348   1 10:06 ?        00:00:00           /usr/bin/ssh-agent /usr/bin/gpg-agent --daemon --sh --write-env-file=/home/jamie/.gnupg/gpg-agent-info-animal /usr/bin/dbus-launch --exit-with-session /usr/bin/monkeysphere-validation-agent /usr/bin/im-launch /home/jamie/.xsession
jamie    28605 28548  0  4915   248   0 10:06 ?        00:00:00           /usr/bin/gpg-agent --daemon --sh --write-env-file=/home/jamie/.gnupg/gpg-agent-info-animal /usr/bin/dbus-launch --exit-with-session /usr/bin/monkeysphere-validation-agent /usr/bin/im-launch /home/jamie/.xsession
jamie    28610 28548  0 22402 34456   0 10:06 tty1     00:00:00           /usr/bin/perl -wT /usr/bin/monkeysphere-validation-agent /usr/bin/im-launch /home/jamie/.xsession

Further down (and not nested under the xinit process), I found my gnome-keyring-daemon process.

I started by removing my own call to gnome-keyring-daemon and realized that everything still worked fine (both gpg and ssh). I must have added that line before things were configured system-wide to run automatically.

Then, I saw that the gpg-agent was being launched without the --enable-ssh-support option (and ssh-agent was running), so looks like ssh is handled by ssh-agent and gpg is handled by gpg-agent. So far so good.

Now, how do they get launched??

I start my graphical session by logging in at a console and running exec startx. In my home directory, I have ~/.xsession which contains the line exec /usr/bin/openbox-session. Therefore, I started by scouring all things openbox on my system and couldn’t find any reference to these agents.

Next, I turned my attention to X. Based on my ps -eFH output, I started with /etc/X11/xinit/xinitrc, which led me to /etc/X11/Xsession which led me to /etc/X11/Xsession.d and presto, I found my answer: a series of shell scripts that modify a global variable STARTUP which is executed by X.

Thanks Debian developers for making it all work out of the box!

Docker networking... private range or not?

Tue, 19 May 2015 12:07:23 +0000

Am I missing something?

I installed docker and noticed that it created a virtual interface named docker0 with the IP address 172.17.42.1. This behavior is consistent with the Docker networking documentation. However, I was confused by this statement:

It randomly chooses an address and subnet from the private range defined by RFC 1918 that are not in use on the host machine, and assigns it to docker0. Docker made the choice 172.17.42.1/16 when I started it a few minutes ago...

It seems like RFC 1918 defines:

10.0.0.0 - 10.255.255.255 (10/8 prefix)
172.16.0.0 - 172.31.255.255 (172.16/12 prefix)
192.168.0.0 - 192.168.255.255 (192.168/16 prefix)

How is 172.17.42.1/16 from the private ranges listed above? Is 172.17.42.1 a potentially public IP address?

GnuCash with python bindings

Wed, 06 May 2015 09:13:58 +0000

Hats off to the GnuCash developers and the Parit Worker Collective for the python bindings to GnuCash. If there is any doubt that free software makes life easier it’s being able to write your own accounting import scripts while reading the original source code.

It’s 2015-05-06 and I’m updating this blog post that I wrote back in 2011 since you no longer have to rebuild gnucash on Debian to get the python bindings. Thanks Debian!

I’ve also published May First/People Link’s python import scripts via git: git clone git://git.mayfirst.org/mfpl/gnucash-import.git

I’m happy to report that the import script now does the following:

Imports May First/People Link members as GnuCash “Customers”
Imports membership dues invoices as GnuCash Invoices
Imports payments and applies them against the Invoices
Imports deleted invoices and puts them in our Unrecoverable A/R account

At this point we have eliminated all double data entry between our membership database and our accounting system. I can’t say that I enjoy bookkeeping yet, but it’s a lot better than before.

So long email, it's been good to know yuh

Thu, 30 Apr 2015 10:00:52 +0000

Yesterday I permanently deleted 15 years of email.

It wasn’t because I didn’t have enough hard disk space to store it. It’s because I decided, after 15 years, that the benefits of keeping all this email did not outweigh the risks. Although I have never had my email subpoenaed, I have had many legal interactions due to my involvement with May First/People Link, some of which were about finding the real identities of May First/People Link members. I’d rather not risk compromising anyone or needlessly exposing my networks. Now I have an Inbox, Sent Box, Trash Box and Detected Spam Box. The Inbox I empty manually and the other boxes are automatically purged on a scheduled basis.

In this age of surveillance it’s sad to see data evaluated based on risk of exposure.

Getting to know systemd

Wed, 20 Aug 2014 10:17:39 +0000

Update 2014-08-20: apcid needs tweaking. See update section below.

Somehow both my regular work laptop and home entertainment computers (both running Debian Jessie) were switched to systemd without me noticing. Judging from by dpkg.log it may have happened quite a while ago. I’m pretty sure that’s a compliment to the backwards compatibility efforts made by the systemd developers and a criticism of me (I should be paying closer attention to what’s happening on my own machines!).

In any event, I’ve started trying to pay more attention now - particularly learning how to take advantage of this new software. I’ll try to keep this blog updated as I learn. For now, I have made a few changes and discoveries.

First - I have a convenient bash wrapper I use that both starts my OpenVPN client to a samba server and then mounts the drive. I only connect when I need to and rarely do I properly disconnect (the OpenVPN connection does not start automatically). So, I’ve written the script to carefully check if my openvpn connection is present and either restart or start depending on the results.

I had something like this:

if ps -eFH | egrep [o]penvpn; then
  sudo /etc/init.d/openvpn restart
else
  sudo /etc/init.d/openvpn start
fi

One of the advertised advantages of systemd is the ability to more accurately detect if a service is running. So, first I changed this to:

if systemctl -q is-active openvpn.service; then
  sudo systemctl restart openvpn.service
else
  sudo systemctl start openvpn.service
fi

However, after reviewing the man page I realized I can shorten if further to simply:

  sudo systemctl restart openvpn.service

According to the man page, restart means:

Restart one or more units specified on the command line. If the units are not
running yet, they will be started.

After discovering this meaning for “restart” in systemd, I tested and realized that “restart” works the same way for openvpn using the old sysv style init system. Oh well. At least there’s a man page and a stronger guarantee that it will work with all services, not just the ones that happen to respect that convention in their init.d scripts.

The next step was to disable openvpn on start up. I confess, I never bothered to really learn update-rc.d. Everytime I read the manpage I ended up throwing up my hands and renaming symlinks by hand. In the case of openvpn I had previously edited /etc/default/openvpn to indicate that “none” of the virtual private networks should be started.

Now, I’ve returned that file to the default configuration and instead I ran:

systemctl disable openvpn.service

UPDATES

2014-08-20: I’ve recently noticed strange behavior when I wake my laptop. Seems to sometimes go right back to sleep. After doing some digging I traced the problem to some customizations I have made to my laptop’s acpid behavior combined with systemd taking over some apci events.

Up to now, I have created my own /etc/acpi files so I have full control over the acpi events. In particular, I don’t want my laptop to suspend when I close the lid. I only want it to suspend when I press the suspend key. And, when it does suspend, I want it to run my own personal suspend script so I can do things like lock the screen and restart tint2.

I’ve found that systemd launches it’s own acpi event monitoring that ignores my /etc/acpid rules (the systemd “unit” that monitors events is called acpid.socket which exists in addition to acpid.service). The systemd reaction to events is controlled by the systemd-logind.service which has a configuration file: /etc/systemd/logind.conf. By default, systemd-logind.service will put my laptop to sleep when the lid is closed and when the suspend button is pushed. systemd seems to get the signal first, putting the laptop to sleep. After I wake it up, acpid gets the signal - so it goes right back to sleep.

Reading man logind.conf helps. I was able to restore my desired behavior by adding these lines to /etc/systemd/logind.conf:

HandleSuspendKey=ignore
HandleLidSwitch=ignore

Then: sudo systemctl restart systemd-logind.service.

New OpenPGP key

Thu, 09 Jan 2014 09:09:11 +0000

After playing around with How secure is my password, I dediced to start changing the handful of passwords I have committed to memory to being very long phrases rather than shorter, concentrated collections of random-seeming characters (another interesting blog post would be to understand the algorythm behind that site but for now I’ve just taken it at face value).

Despite my usual practice of introducing new passphrases by changing my laptop login (which can always be reset if necessary), I went ahead and changed by GPG key instead to a new 50 character phrase. Fifteen minutes passed before I needed access to the key and… I couldn’t repeat the passphrase.

I tried to be clever. I opened a text file and typed in this passphrase over and over again, ultimately generating 49 unique typo-ridden versions of the passphrase.

Then, I wrote a simple script that took each of these versions and generated additional versions by:

Changing one character (from a preselected character list of letters, numbers and punctuation within key range of the actual letters in the passphrase).
Deleting one character
Adding one character
Transposing all characters

In the end I had about 350,000 versions of my passphrase.

None of them worked.

What in the world did I type??

Could I just ask the NSA?

In any event, my new OpenPGP key fingerprint is:

1F9C30CB3CFC5DA9987FA035A014C05A607B7535

Please note: I’ve changed the “real name” part of my User Id from “Jamie McClelland” to “James McClelland” becuase James is on my government issued id.

I’ll be working on collecting new signatures for the rest of my life.

Some interesting information on how this made my life difficult:

I use the Monkeysphere for accessing the servers I maintain using by OpenPGP key. Fortunately, the key was loaded into my ssh agent, so I have still been able to access the servers. If my computer crashes or gets restarted I’ll be locked out until I get my new key in place. I’ll need to ask another May First/People Link administrator to sign a puppet release with my new fingerprint.
May First/People Link uses keyringer to store shared passwords, encrypted to a small collection of administrator OpenPGP keys, including my old one. I’ll need to ask another admin to update the list of keys.
I have hundreds of (mostly) web-site passwords stored in assword and encrypted to my old key. They are all gone. I’ll be doing lots of password resets now.
I OpenPGP sign my emails. That has been turned off between the time I lost the passphrase and generated a new key.
I am subscribed to a few email lists run via Schleuder. I can’t access those emails now. I’ll need to send my new key to the list.
I’ve lost access to every encrypted message that has been sent to my old key.

Postscript: Thanks to Jordan Uggla, I was directed to passe-partout, a tool that can extract an RSA private key out of memory when ssh-agent is running with the key loaded. Since I am using the Monkeysphere, my OpenPGP key has an authentication-capable subkey, which is fed to an ssh-agent and used to grant me access to remote computers using ssh. After losing my passphrase, I was not able to access that subkey. With the help of pass-partout, and the fact that the subkey was loaded into ssh-agent before I lost the passphrase, I was able to extract the private RSA key from my ssh-agent and save it to a file. Then, I could restart my ssh-agent with my ‘’new’’ authentication-capable OpenPGP subkey, thus allowing me to access servers that have granted me access via my new key and the handful of servers that still only grant me access via my old key.

Identities

Wed, 08 Jan 2014 09:32:44 +0000

My OpenPGP and SSH public keys are available below.

OpenPGP

Import from the public key servers:

gpg --keyserver keys.openpgp.org --recv-key 1F9C30CB3CFC5DA9987FA035A014C05A607B7535

Or, click to download my public GPG key.

SSH public key

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDiGm0ZoC/lgBDfyeRKcciXQWsJ/odB3zOS/8YMpTCMy8bPbRIZwNvOlIh6EG17bWzAKfMKoivY9QncXjtUeFBq4lDqxuxe2UU29Pwux01jdU+QOZ73M65/hVMFYEkV5qJAc8yf2WNq9vX472eIE6rQlo4DmtNJnuGvoQpF51ZV3uZUqc9QmT9+Vwg3RaOChKzTG3beE+SCJkrMxsMq0jhAT4BR1Xoknyo20Gmr3F7l84Un7U5YwFTRHNKs32nXv0Yhfdd7qhxZLyB4gQx+pVq/0kYCyjzhb2AZmmtLcAs7Btop045k1fbjmOkxnWDqw8XX+AAaNsNn85a8GZU1GcRn MonkeySphere2014-01-08T14:39:16 James McClelland <jamie@mayfirst.org>

Jamie's Bio

Wed, 08 Jan 2014 09:32:44 +0000

Jamie is technology systems director for the Progressive Technology Project, a national organization providing technology support to grassroots organizing groups in the US.

Jamie is also co-founder and Board member of May First Movement Technology, a membership organization of progressive groups worldwide who use the Internet. In his work with May First, Jamie does political organizing, systems administration, and support for the members of May First.

May First is the result of a merger in 2005 between May First Technology Collective (originally known as Media Jumpstart) and People-Link. Prior to the merger, Jamie was co-founder and co-director of May First Technology Collective, a worker run nonprofit organization that provided technical support to NYC’s social justice movement groups.

Prior to working at May First, Jamie worked at Libraries for the Future as network administrator, national Youth ACCESS coordinator, and Information and Technology Policy Specialist. Jamie was formerly on the Board of Directors of Paper Tiger TV where he was an active producer and activist between 1994 and 2004. Previously, Jamie worked as a video instructor for Sidewalks of New York, teaching basic production skills to homeless youth. He also worked as a community organizer for the Association of Community Organizations for Reform Now (ACORN) and was an active member of ACT UP New Orleans.

You can reach jamie at jamie @ workingdirectory dot net.

Administering CUPS from the command line

Tue, 08 Jan 2013 09:32:44 +0000

I usually try to avoid administering printers whenever possible. As a result I end of flailing around the CUPS web interface before I figure out how to re-enable a printer. And, when I get a call to help debug a printer, I can’t easily tell people what to do.

When I try to do what I need via the command line, I end up spending at least 10 or 15 minutes re-reading man pages before I piece together the steps.

Here’s my attempt to document the steps so I don’t have to re-read man pages.

Setup

The cups commands in these examples can be run as a non-root user if that user is in the lpadmin group.

Type:

groups

To see if lpadmin is listed. If not:

sudo adduser <your-user-name> lpadmin

Then, to gain access to the new group without logging out and logging in again:

newgrp lpadmin

Finding printers.

In these examples, the printer name in question is: stability and it is a network printer, with local DNS that properly resolves the hostname stability to an IP address.

You can list all detected printers with: lpstat -v

Note that the resolved IP address is listed if it’s a network/wifi connected printer.

Network access

First, try to ping the printer:

ping stability

If this fails, restart the printer and/or check network cables. No point in doing anything else until it responds to pings.

Can’t submit new jobs to the printer

Next, if the problem is that the printer is greyed out when you try to print a document or your application tells you that the printer is rejecting jobs, confirm this status with:

lpstat -a stability

It will either output:

stability accepting requests since Mon 20 May 2013 10:28:57 AM EDT

stability not accepting requests since Mon 20 May 2013 10:28:57 AM EDT -
  Rejecting Jobs

If it is rejecting jobs, try:

/usr/sbin/cupsaccept stability

Accepts new jobs, but just doesn’t print

On the other hand, if the printer is accepting jobs, but the jobs are not printing, find out if the printer is enabled with:

lpstat -p stability

You should get either:

printer stability is idle.  enabled since Mon 20 May 2013 10:28:57 AM EDT

Or:

printer stability disabled since Mon 20 May 2013 10:35:10 AM EDT -
  Paused

If it is disabled, you should first see what queued jobs there are:

sudo rlpq -P <printer> -H <printer>

(If this fails, try replacing the -H option with the IP of the printer - remember, run lpstat -v to see the IP address of your printer.)

If you have a list of duplicate pending jobs, be sure to delete the duplicates to avoid having your print job come out multiple times.

To delete a queued job, type the following (n should be the number in the Job column of the lpq output):

cancel <n>

After you have deleted duplicate jobs, try “enabling” it:

/usr/sbin/cupsenable stability

Then, re-rerun the lpq command and see if it’s now “ready.” At this point, the jobs should start printing.

Submit a job via the command line

lp -h <printer-host> /path/to/file
lp -d <printer-name> /path/to/file

If a pdf doesn’t print, try “fixing” the PDF with:

mutool clean input.pdf output.pdf

Review of concepts

For review… a few important concepts:

cupsaccept/cupsreject: controls whether a printer will accept or reject new jobs. It doesn’t matter whether the printer is enabled or disabled.
cupsenable/cupsdisable: controls whether a printer will print existing jobs. It doesn’t matter whether the print is accepting or rejecting new jobs.

If you don't know what a quine is, consider yourself lucky

Tue, 08 Jan 2013 09:32:44 +0000

rhatto just asked me to approve a change in keygringer’s license from AGPLv3+ to GPLv3+ citing a discussion about a Berkeley DB’s switch to AGPLv3. For some reason, a reference to a quine caught my eye.

If you don’t know what a quine is, I suggest you remain ignorant. I won’t even provide the Wikipedia link.

A quick web search suggests that it’s quite possible to write one in bash. I hope to be productive again some day.

Sharing screen with remote Linux user behind a firewall

Tue, 08 Jan 2013 09:32:44 +0000

Intentionally helping someone poke a hole through their firewall to allow any users on a remote machine access to their computer is generally a bad idea… unless they want you to help them with their computer.

In that case, I find it really useful.

Toward this end, I’ve setup a dedicated user account (jamie-share@chavez.mayfirst.org in this example). I granted myself shell access to this user and generated an ssh key pair:

ssh-keygen -t rsa

Then, as a convenience I move my public key into the home directory:

cp ~/.ssh/id_rsa.pub ~/

Next, I give the user I want to help ssh access to this shell account on an Internet connected computer (in this example: jamie-share@chavez.mayfirst.org). Either I share the password with them or if they have an ssh key or monkeysphere identity I use that.

Then, I ask them to grant teh jamie-share user access to their user account by running (which downloads the key I just created):

mkdir -p ~/.ssh
ssh jamie-share@chavez.mayfirst.org "cat id_rsa.pub" >> ~/.ssh/authorized_keys

Next, ask them to install openssh-server on their local computer:

sudo apt-get install openssh-server screen

Next, they run the following command on their local computer:

ssh -R 2222:localhost:22 jamie-share@chavez.mayfirst.org

This command says: forward port 2222 on chavez.mayfirst.org to port 22 on your local computer.

Lastly, I log in to jamie-share@chavez.mayfirst.org and run, replacing with their local username.

ssh <their-username>localhost -p 2222

You can do whatever you want now, but running screen is a good way to share a session so the person you are working with can see what you are doing as you do it.

p.s. Thanks Ross for the tips!

Creating virtual machines

Sun, 08 Jan 2012 09:32:44 +0000

I recently setup an Alix board from PC Engines. I installed Debian onto a Compact Flash card using grml for the first time.

After using parted to create a single partition, I ran:

grml-debootstrap --arch i386 --filesystem ext4 --grub /dev/sdb --hostname yaqeen --mirror http://http.us.debian.org/debian \ 
--nopassword  --release squeeze --target /dev/sdb1

I had to manually mount the partition and add console redirection for grub (and /etc/inittab).

And then, when booting, I had to manually edit the grub configuration because it was trying to find the root filesystem on /dev/sdb instead of /dev/sda. I’m sure there is some tweaking I can do via /etc/deboostrap/scripts to fix those needs.

Overall, I was very happy with the results. Thanks grml developers!

Update 2012-06-19

When installing a server I prefer keeping things a bit more manual.

I start by doing a pxeboot into a debirf image. Then, I run through all the disk partitioning/encrypting/etc steps listed on the May First/People Link installation page.

Then, I mount each partition in /mnt and run debootstrap:

debootstrap squeeze /mnt

Copy over any non-free firmware packages, e.g.:

cp /media/usb/firmware-bnx2_0.28+squeeze1_all.deb /mnt/root/

Next, mount the system directories and chroot:

for dir in sys dev proc; do mkdir -p /mnt/$dir; mount -o bind /$dir /mnt/$dir; done
chroot /mnt

Install non-free firmware packages, e.g.:

dpkg -i /root/firmware-bnx2_0.28+squeeze1_all.deb

Add /etc/fstab, /etc/network/interfaces, /etc/crypttab files

Edit /etc/inittab to enable output to the serial console

Set the root password.

Set the hostname (edit /etc/hostname)

Install necessary packages that debootstrap won’t install for you:

apt-get update
apt-get install linux-image-2.6-amd64 lvm2 cryptsetup mdadm grub-pc

When prompted by grub for Linux command line options, enter:

console=ttyS0,115200n8

Edit /etc/default/grub adding to the bottom:

GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1"

Edit /boot/grub/device.map to remove any USB devices

And run update-grub

Key Mapping for console redirection

Sun, 08 Jan 2012 09:32:44 +0000

This always flashes by too quickly for me to read…

I’m preserving it here for next time.

Press the spacebar to pause...

KEY MAPPING FOR CONSOLE REDIRECTION:

Use the <ESC><0> key sequence for <F10>
Use the <ESC><!> key sequence for <F11>
Use the <ESC><@> key sequence for <F12>

Use the <ESC><Ctrl><M> key sequence for <Ctrl><M>
Use the <ESC><Ctrl><H> key sequence for <Ctrl><H>
Use the <ESC><Ctrl><I> key sequence for <Ctrl><I>
Use the <ESC><Ctrl><J> key sequence for <Ctrl><J>

Use the <ESC><X><X> key sequence for <Alt><x>, where x is any letter
key, and X is the upper case of that key

Use the <ESC><R><ESC><r><ESC><R> key sequence for <Ctrl><Alt><Del>

Killing MySQL

Sun, 08 Jan 2012 09:32:44 +0000

MySQL is supposed to allow one + the max_connections from a super user so you can stop/restartmysql even when it is reporting too many connections. Unfortunately, for reasons I’m not entirely sure of, sometimes that connection seems to be in use.

I’ve encountered this problem with web apps and now follow these steps:

Shutdown apache to prevent new connections
Search for and kill any backup processes (this can cause table locking which can be the cause of the problem)
Try a clean shutdown again
search for and kill mysqld_safe safe process on host. This step should kill one connection allowing you to cleanly stop mysql with:

/etc/init.d//mysql stop

Live video streaming using vp8enc and webm on Debian

Sun, 08 Jan 2012 09:32:44 +0000

The vp8 video codec and webm container format (which are alternates for theora/ogg and h264/flv) are getting lots of attention, especially as more web browsers are supporting html5 - which includes a new video tag that plays video directly in a web page, rather than requiring a plugin. With Google fully behind these new formats (and converting YouTube videos) and even skype using vp8, it seems as though it could supplant h264/flash as the default video codec and container.

After years under the boot of the proprietary flash video, this change could usher in a lot of exciting developments.

While the shift in formats is also significant for on-demand video, I’m focusing on live streaming in this blog.

There are a lot of tools needed to successfully stream a live video. Enough of them support vp8 and webm to create a live video stream, however, not all of the right versions of these tools have landed in Debian.

Without any patches or changes, if you try to send a live video stream to icecast using vp8 encoding and webm, you’ll get something like this:

0 jamie@animal:~$ gst-launch v4l2src ! video/x-raw,width=320,height=240 ! vp8enc ! webmmux ! shout2send ip=icecast.server port=8000 password=secret mount=/test.webm
WARNING: erroneous pipeline: could not link webmmux0 to shout2send0
1 jamie@animal:~$

I’ve documented the steps for patching/upgrading gstreamer and icecast2 so you can get a live video stream using vp8 encoding and the webmc container that originates on a Debian wheezy machine using gstreamer and streams to a Debian squeeze server running icecast2.

On the workstation, you’ll need libshout3 version 2.3.0 or higher, which is at the moment available in experimental and installable without bringing in any dependencies. In addition, you will need to bring in gstreamer1.0 from sid and you will need to rebuild gstreamer1.0-plugins-good after libshout-dev has been installed from experimental.

So, begin by adding experimental and sid to your sources list by creating the files /etc/apt/sources.list.d/experimental.list and /etc/apt/sources/lists.d/sid.list with the respective contents:

deb http://mirror.cc.columbia.edu/debian experimental main

and

deb http://mirror.cc.columbia.edu/debian sid main 
deb-src http://mirror.cc.columbia.edu/debian sid main

In order to avoid upgrading your entire machine to sid (experimental packages won’t upgrade by default), you’ll also need to add the file /etc/apt/preferences.d/sid:

Package: *
Pin: release n=sid
Pin-Priority: 200

Then update and install:

apt-get update
apt-get install -t experimental libshout3
apt-get install gir1.2-clutter-gst-1.0 gir1.2-gst-plugins-base-1.0 gir1.2-gstreamer-1.0 gstreamer0.10-ffmpeg  gstreamer1.0-libav:amd64 gstreamer1.0-plugins-bad:amd64 gstreamer1.0-plugins-base:amd64 gstreamer1.0-plugins-base-apps gstreamer1.0-plugins-good:amd64 gstreamer1.0-pulseaudio:amd64 gstreamer1.0-tools gstreamer1.0-x:amd64 libclutter-gst-1.0-0:amd64 libgstreamer-plugins-bad1.0-0:amd64 libgstreamer-plugins-base1.0-0:amd64 libgstreamer-plugins-base1.0 libgstreamer1.0-0:amd64 libgstreamer1.0

Fortunately, the version of gstreamer1.0-plugins-good in sid includes a fix for a bug patched on November 31, 2012 that allows the use of the webm container. However, for it to work properly gstreamer1.0-plugins-good has to be built against libshout version 2.3.0 or higher (which is in experimental, not sid). Therefore, you have install libshout-dev from experimental and then rebuild gstreamer1.0-plugins-good.

sudo apt-get build-dep gstreamer1.0-plugins-good
sudo apt-get -t experimental install libshout-dev
apt-get source gstreamer1.0-plugins-good
cd gst-plugins-good1.0-1.0.4
dch -v 1.0.4-1+webmmux
fakeroot debian/rules binary
sudo dpkg -i ../gstreamer1.0-plugins-good_1.0.4-1+webmmux_amd64.deb

Now if you try the command, things should work on the client side. When streaming to an icecast2 server running squeeze, I don’t get any errors, however, when opening the streaming page via the icecast URL, there is a play button, which can be clicked, but no picture is shown. That’s because the version of Icecast2 in squeeze doesn’t support either vp8 or webm.

On the server side, you will need icecast2 version 2.4beta or higher. This version hasn’t landed anywhere in Debian as of this writing.

However, you can build it from source by following these steps (the patches I’m removing have all been applied to the new upstream version).

apt-get source icecast2
wget http://downloads.xiph.org/releases/icecast/icecast-2.4-beta.tar.gz
tar -xzvf icecast-2.4-beta.tar.gz
cp -r icecast2-2.3.2/debian icecast-2.3.99.0/
cd icecast-2.3.99.0/
rm debian/patches/{1001_autotools_avoid_debian-subdir.patch,1002_add_missing_xspf_file,1003_fix_memory_leak.patch,series}
dch -v 2.3.99.0-1
fakeroot debian/rules binary

When you install it, be sure to replace the icecast2.xml configuration file and update it with your server-specific customizations.

Hopefully this version of icecast2 will make it to experimental soon.

And now… you can successfully stream via:

gst-launch-1.0 v4l2src ! video/x-raw,width=320,height=240 ! vp8enc ! webmmux !  shout2send ip=icecast.server port=8000 password=secret mount=/test.webm

If you want full audio and video, here’s a more complete pipeline:

gst-launch-1.0 v4l2src ! queue ! video/x-raw,width=320,height=240 ! tee name=tscreen ! queue ! autovideosink tscreen. ! queue ! videorate ! video/x-raw,framerate=25/2 ! queue ! vp8enc ! queue ! webmmux name=mux pulsesrc ! queue ! audioconvert ! vorbisenc ! queue ! queue ! mux. mux. ! queue ! shout2send ip=hobo port=8000 mount=jamie.webm password=secret

Unfortunately, since I’m not sure how to deal with the python2.7 vs python3 dependecies, my elegant mobi python script that creates this pipeline for you using python-gtk as been reduced to a crude bash script.

Another item on the todo list is opus audio encoding. gstreamer doesn’t seem able to use opus in the webm container (although it seems to work with ogg).

In the course of this research I found directions for getting gstreamer-1.0 installed on debian independent of packages installed on the system. If you follow the directions above, these steps should not be necessary. I’m including in case you need the latest sources for another reason.

Extra steps I needed were:

Edit gst-rebuild, comment out gst-plugins-ugly from plugins var (line 14) and sources arg (line 14) - these caused errors for me.
manually install the following packages from experimental (apt-get build-dep gets confused about how to find them):
apt-get install -t experimental libavcodec-dev=6:9~beta3-1
apt-get install -t experimental libavformat-dev=6:9~beta3-1
apt-get install -t experimental libswscale-dev=6:9~~beta3-1 libswscale2=6:9~~beta3-1
Then follow directions as advertised

Managing KVM instances

Sun, 08 Jan 2012 09:32:44 +0000

At May First/People Link we have been using KVM for several years now and recently I have been running KVM instances on my local laptop.

I’m pleased to see all the work that has gone into libvirt, which seems like a robust and full-featured suite of tools for managing many virtualization technologies, including KVM. However, we don’t use it at May First/People Link for a number of reasons. The most pressing is that it runs all virtual guests as the same user, but also because it offers far more features than we need (such as graphical access to virtual server, which we don’t need since none of our guest servers run X).

On May First/People Link hosts, we are using a relatively simple set of bash scripts (accessible via git at git://lair.fifthhorseman.net/~dkg/kvm-manager). These scripts re-use many tools we are already familiar with to build and launch kvm guests. Each guests runs as a dedicated non-provileged user, with a console available using screen, and the kvm process is managed using runit. Since our admins are familiar with these tools already, the learning curve involved is much less steep.

Despite the relative simplicity of kvm-manager, it was still more complicated and involved than I wanted on my laptop. Additionally, I wanted to fully understand every piece of the puzzle and separating out user privileges wasn’t important to me.

So - I wrote the a bash script to launch virtual servers. It assumes you are using logical volume manager.

Some editing required if you want to re-use it. You can presuse it below or download it.

# !/bin/bash
# Manage a virtual server

# This script assumes you are using Logical Volume Manager (LVM)
#
# There are changes to your system that you need to make once to get this
# system working. Once you have made these changes you are done.  There are
# other steps you have to take everytime you add a new guest to your
# system.
#
# ONE TIME CHANGES
#
# Install necessary packages 
#
# sudo apt-get install qemu-kvm screen bridge-utils dnsmasq
#
# For networking to properly work, your kernel must allow packet forward.
#
# You can enable packet forwarding by adding the file /etc/sysctl.d/local.conf
# with the contents:
#
# # used for networking kvm instance
# net.ipv4.ip_forward=1
#
# When you restart your computer, this change will effect. Or, you can run the
# following command to get it to take effect right away:
#
# sudo -i    # to become root
# echo 1 > /proc/sys/net/ipv4/ip_forward
# exit    # to return to being a normal user
#
# You must modify your /etc/network/interfaces file and add the following
# stanza:
#
# auto virbr0 
# iface virbr0 inet static 
#  address 10.11.13.1 
#  netmask 255.255.255.0 
#  pre-up brctl addbr virbr0 
#  post-down brctl delbr virbr0
#
# Then, bring up the interface with:
# 
# sudo ifup virbr0
# 
# Next configure dnsmasq by creating the file: /etc/dnsmasq.d/local with the following
# content:
#
# interface=virbr0 
# dhcp-range=10.11.13.2,10.11.13.100,1h
#
# Restart dnsmasq for the changes to take place:
#
# sudo service dnsmasq restart
#
# In order to run your virtual server as a non-privileged user (e.g. your
# normal user) you will need to make a change to your system so that your
# newly created logical volume (and all future logical volumes) will be
# owned by your user. 
# 
# Add a file called /etc/udev/rules.d/92-kvm.rules with the following line
# (change "jamie" to the group you are running as and vg_kermit0 to the
# name of your volume group). If you are not sure the name of your volume
# group type: sudo vgs.
#
# ACTION=="change", SUBSYSTEM=="block", ATTR{dm/name}=="vg_kermit0-*_root", GROUP="jamie"
#
# Lastly, you will need to download a Debian installer ISO to your file
# system.
# 
# Example command to download the full installer (CD 1):
# wget http://cdimage.debian.org/debian-cd/6.0.5/amd64/iso-cd/debian-6.0.5-amd64-CD-1.iso
# 
# Example command to download the net installer (smaller download, will use 
# the Internet to download needed packages):
#
# wget http://cdimage.debian.org/debian-cd/6.0.5/amd64/iso-cd/debian-6.0.5-amd64-netinst.iso
#
# REPEAT ONCE FOR EACH GUEST
#
# Now, create a new logical volume for your virtual server.  You need to
# repeat this step for every virtual server you create. This example
# assumes that you are creating a virtual server named gonzo and that your
# logical volume group name is vg_kermit0 (if you are not sure what your
# logical volume group is named, try typing the command: sudo vgs).
#
# sudo lvcreate --name gonzo_root --size 15GB vg_kermit0
#
# Finally, assuming your ISO is stored in /home/jamie/ISOs/debian.iso,
# type: 
#
# ./vlaunch gonzo start /home/jamie/ISOs/debian.iso 
#
# and you are ready to go. 
# 
# This form of the command will start your virtual server with the Debian
# installer passed to it and you should be prompted through the
# installation.
#
# After you have installed Debian, you can start it with simply:
#
# vlaunch gonzo
#
# If you want to clean up the networking devices created (after you have
# shutdown your virtual server) you can do that with:
#
# vlaunch gonzo cleanup
#
# Be sure to edit the variables below to match your system:

bridge=virbr0
user=jamie
vg=vg_animal0
# Change graphic to 0 if you want to launch this via screen
graphic=1
# Change configure_nat to 0 if you want to handle your nat creation
# on your own (e.g. via /etc/network/ifup.d/
configure_nat=1

# Modify server memory here. Depending on how much memory you have for your entire
# system you may want to raise or lower this number
mem=512

# Create a function that will echo a variable passed in and then exit the script
die () {
  printf "$1\n"
  exit 1
}

# This is the function that will be called if we are starting a virtual server
function start() {
  if [ "$configure_nat" -eq 1 ]; then
    # Get the name of the current network device
    dev=$(ip route | grep ^default | grep -oE "dev [a-z0-9]+" | sed "s/dev //")

    if [ -n "$dev" ]; then
      # Flush the nat table to avoid duplicates
      sudo iptables --table nat -F

      # Create a NAT (network address translation rule)
      sudo iptables --table nat -A POSTROUTING ! -d 127.0.0.1/8 --out-interface "$dev" -j MASQUERADE
    else
      printf "I could not determine your network device. Not configuring NAT.\n"
    fi
  fi
  lvname="${vg}-${server}_root"

  # Trigger udev to ensure we have proper ownership of the block device.
  sudo udevadm trigger --subsystem-match=block --attr-match=dm/name="$lvname"

  lv="/dev/mapper/$lvname"
  [ ! -e "$lv" ] && die "Can't find $lv"

  sudo modprobe -v tun || die "Failed to modprobe tun module"

  # Create network device if it doesn't already exist.
  ip tuntap | grep "$tap" >/dev/null || sudo ip tuntap add dev "$tap" mode tap user "$user" || die "Failed to create device $tap"
  # Bring up device if it's not already up.
  ip link | grep " $tap " >/dev/null || sudo ip link set "$tap" up || die "Failed to set $tap to up"
  # Add the device to the bridge so it get use the upstream network connections.
  /sbin/brctl show | grep "$tap" > /dev/null || sudo brctl addif "$bridge" "$tap" || die "Failed to add tap to bridge"

  # Launch kvm.
  screen=
  nographic=
  if [ "$graphic" -eq 0 ]; then
    # Launch with -nographic in a screen session
    screen="screen -S $server"
    nographic="-nographic"
  fi

  if [ "$command" = "show" ]; then
    printf "Here is the command that would be executed:\n"
    echo kvm -drive "file=$lv,if=virtio,id=hda,format=raw" -m "$mem" -device "virtio-net-pci,vlan=1,id=net0,mac=$mac,bus=pci.0" -net "tap,ifname=$tap,script=no,downscript=no,vlan=1,name=hostnet0" $cdarg $nographic 
  else
    $screen kvm -drive "file=$lv,if=virtio,id=hda,format=raw" -m "$mem" -device "virtio-net-pci,vlan=1,id=net0,mac=$mac,bus=pci.0" -net "tap,ifname=$tap,script=no,downscript=no,vlan=1,name=hostnet0" $cdarg $nographic || die "Failed to start kvm" 
  fi
}

# This is the function we will call to cleanup.
function cleanup() {
  read -p "Please shutdown the host first then hit any key to continue..."
  sudo brctl delif "$bridge" "$tap"
  sudo ip link set "$tap" down
  sudo ip tuntap del mode tap dev "$tap"
}

# Start the main program logic.
# The first argument passed to the script is $1 - resave as the variable $server
server="$1"

# Second argument is the command
command="$2"
# If no command is passed, assume we are starting.
if [ -z "$command" ]; then
  command=start
fi

# By default cdarg variable is left empty
cdarg=

# If a third variable is passed, it means they are passing an ISO image.
if [ -n "$3" ]; then
  # Make sure it exists.
  [ ! -f "$3" ] && die "Third argument should be path to cd iso. Can't find that path."
  cdarg="-cdrom $3"
fi


# Generate reproducible mac address.
mac="$(printf "02:%s" "$(printf "%s\0%s" "$(hostname)" "${server}" | sha256sum | sed 's/\(..\)/\1:/g' | cut -f1-5 -d:)" )"
tap="${server}0"

if [ "$command" = "start" ] || [ "$command" = "show" ]; then
  start "$command"
elif [ "$command" = "cleanup" ]; then
  cleanup
else
  die "Please pass start or cleanup as first argument. You passed: $command"
fi

MySQL throwing the wrong error

Sun, 08 Jan 2012 09:32:44 +0000

I was running a routine CiviCRM installation when I was stopped by the error:

ERROR 1071 (42000) at line 2415: Specified key was too long; max key length is 1000 bytes

The line referred to a table that created two indexes. Each index combined a 512 byte varchar field with a 4 byte integer field (well below 1000 bytes).

I then tried downloading the sql file to my local computer and it imported without a problem.

So… I started the painstaking process of comparing the configurations on each computer. Eventually, I came to this setting that was causing the problem:

innodb_log_file_size = 268440000

In fact, it wasn’t the setting causing the error. The error was caused because I added that setting without properly shutting down MySQL, removing /var/lib/mysql/ib_logfile0 and /var/lib/mysql/ib_logfile1 files and then restarting MySQL.

New Privacy Arguments are Needed

Sun, 08 Jan 2012 09:32:44 +0000

Another decent article on online privacy was published by the NYT this morning. The article documents the collection and centralized aggregation of consumer data online. It ends with a strong message about consumers becoming the “product” - which echoes what Noam Chomsky and Ben Bagdikian have been saying for decades.

The problem with this critique is that it doesn’t go far enough. Chomsky and Bagdikian didn’t say readers are the product of modern journalism and that’s bad because nobody wants to be a product. They critiqued the model because it produces terrible journalism and democracy suffers as a result.

Similarly, the online trading of audiences and the invasion of our privacy that results has more important implications than “don’t be a sucker” which seems to be the gist of the privacy argument these days. Honestly, I don’t stay awake at night worrying that Nike knows I’m looking for a pair of sneakers, or even that they know my income.

I do stay awake at night worrying that people are tagging my photo on Facebook, which could allow the New York Police Dept to submit a photo of protesters to Facebook and get a list of names and addresses of the people in the photo. Or it could allow the police to track my movements via existing networks of surveillance cameras by matching my image to my name. Would that require a search warrant? How would that impact my trust in my government to know that my movements are being tracked? Or worse, to know they might be tracked but I’ll never know if they are or aren’t?

Many governments (including the US) have sordid histories of infiltrating political organizations and intentionally sowing dissent. Having access to activists’ purchasing habits provides a treasure trove of material for this purpose that previously was labor intensive for cops to collect. It also provides leverage during interrogation, a time of heightened emotional vulnerability where the mere mention of a private detail of one’s life can have a powerful impact. How can we prevent law enforcement from having access to these details of our lives?

And then there’s the dragnet. A central database of consumer habits provides a tantalizing collection of information that could be searched for people who fit a profile of a crime. Is solving the crime at hand worth the invasion of privacy that results from people being investigating only based on their consumer habits? How would we measure the impact an investigation like this would have on our confidence in democracy and government?

The corporate advertisers promoting this kind of data collection argue that these problems are the price of progress. However, the ideology behind these technology developments is capitalism, not some intrinsic aspect of the Internet or network communications. Historically, the Internet has developed based on a drive toward de-centralization. Email and web sites, the bedrock of Internet activity, are based on these principles: anyone can add their own email server or web server and everyone can seamlessly access it. In contrast, the logic of capitalism as it relates to the Internet, is one of centralization and aggregating data. Facebook and Google want you to use their platforms for all your online activities so they have as complete a picture as possible of what you like and do. As the New York Times article describes, companies like Rubicon want to track all your movements and aggregate that data for profit.

We have a choice. We can carefully evaluate our use of corporate services and consider the implications it has not only on our lives but on society as a whole. And, we can choose to use and support services that promote open standards and interoperability that will respect our personal privacy values.

Want a practical example? Friendica is software designed to allow you to post status alerts, photos, videos and other information about yourself in a way similar to Facebook. However, it has one big difference: you can host your account on any server running the software anywhere in the world. You can still “friend” people on different servers, but we don’t have to all trust the same organization to host it. Furthermore, Friendica allows you to post once and automatically cross-post to your Twitter, Facebook or other similar services. Sound good? You are welcome to try out an account on the May First/People Link install: https://friends.mayfirst.org/.

Posting to identi.ca via the command line

Sun, 08 Jan 2012 09:32:44 +0000

I set aside 15 minutes to find a tool that would allow me to easily post to my identi.ca account via the command line. I’m two hours in and I finally sent my first remote post. Hopefully nobody else will have to spend this much time!

There are a lot of tools to help you interact with Twitter via the command line and identi.ca supports the twitter API, however, convincing these tools to use identi.ca was harder than I expected and is woefully under-documented. This hardship is largely self-inflicted since I chose to authenticate via oauth, however, it’s not clear to me if identi.ca supports basic auth and if so for how much longer. In any event, oauth seems like a much preferable authetication approach because I don’t have to store my identi.ca password in plain text.

I decided to use tweepy since it’s packaged for Debian and it’s written in python.

If you are not familiar with OAuth, it’s worth reading up on. In short:

register your application via identi.ca web interface
retrieve consumer token and consumer secret from identi.ca (via the web)
using this token and secret, request from identi.ca an application key and secret (you must provide your identi.ca user login credentials before you get these strings)
and now, configure your application to use the applicaiton key and secret everytime it connects

tweepy has documentation on using setting up oauth, however, I found it hard to follow, especially since it goes back and forth between using it for a web app and using it for a desktop app. Since I’m interested in a desktop app (shell), the web app business was just clutter.

I wrote the script below to initialize my app and retrieve the application key and secret. If you are writing a desktop app, this needs to be run once for each user of the app:

#!/usr/bin/python

""" 
This script should be used once to initialize your desktop/command line
application with a oauth access key and access secret. 

Your first step is to login, via the web, to your identi.ca account
and authorize your application. Click Edit next to your username
to edit your profile settings. Then click Connections on the left
side bar. Then, in the right sidebar, click 
"Register an OAuth client application".

When you are done, you will see a page listing your token and secret.
Fill in the consumer_token and consumer_secret variables below with those
values.
"""
""" Fill in these values with values provided in the identi.ca web app when you register your app! """
consumer_token = ""
consumer_secret = ""

import tweepy

host = 'identi.ca'
api_root = '/api/'
oauth_root = api_root + 'oauth/'

auth = tweepy.OAuthHandler(consumer_token, consumer_secret, 'oob')

auth.OAUTH_HOST = host
auth.OAUTH_ROOT = oauth_root
auth.secure = True

try:
  redirect_url = auth.get_authorization_url()
except tweepy.TweepError:
  print 'Error! Failed to get request token.'
  quit()

req_key = auth.request_token.key
req_secret = auth.request_token.secret

print "you don't need these values... just fyi..."
print "auth request key is: " + req_key
print "auth request secret is: " + req_secret

print "Go to this URL for verify code: " + redirect_url

print "enter the verify code from the URL above"
verifier = raw_input('Verify code: ')

auth = tweepy.OAuthHandler(consumer_token, consumer_secret, 'oob')
auth.set_request_token(req_key, req_secret)

auth.OAUTH_HOST = host
auth.OAUTH_ROOT = oauth_root
auth.secure = True

try:
  auth.get_access_token(verifier)
except tweepy.TweepError:
  print 'Error! Failed to get access token.'

print "Store these values in your application. You will re-use them"
print "auth access key is: " + auth.access_token.key 
print "auth access secret is: " + auth.access_token.secret 
print "done with initialization"

Once you have your key and secret, the following simpler application will post for you:

#!/usr/bin/python
import tweepy

post = "I microblog from bash"
consumer_token = ""
consumer_secret = ""
access_key = ""
access_secret = ""

host = 'identi.ca'
api_root = '/api/'
oauth_root = api_root + 'oauth/'

auth = tweepy.OAuthHandler(consumer_token, consumer_secret, 'oob')
auth.OAUTH_HOST = host 
auth.OAUTH_ROOT = oauth_root 
auth.secure = True 
auth.set_access_token(access_key, access_secret)

api = tweepy.API(auth, host = host, api_root = api_root)
api.update_status(post)

Property is Theft

Sun, 08 Jan 2012 09:32:44 +0000

I’ve recently been making my way through the Proudhon Reader, a collection of writings by Pierre-Joseph Proudhon recently published by AK Press. Proudhon, known as the father of Anarchism is most famous for his declaration: “Property is theft.”

The simple declaration is based on pages of explanation which, simplistically, can be boiled down to the idea that workers should all equally share in the fruits of their labor. While Proudhon’s influence on Marx is hotly debated, this idea is fundamental to Marx’s idea around the alienation of labor. If we are working to produce a product that is then owned by the proprietor, rather than the worker, we are alienated from our labor.

Many May First/People Link members, such as Palante Tech, Openflows, Union Web Services, and Agaric Design are all committed to fighting this alienation by organizing as worker-run cooperatives. Nobody is the boss, all workers share in all profits.

However, Proudhon got me thinking about free software shops that aren’t worker-run cooperatives. If you are employed by a for-profit corporation as a worker and your only job is to write code that is released under a free software license, are you alienated from your labor in the Marxist sense? The fruits of your labor might be owned by your company, but they are freely licensed to the world (which, of course, includes you).

There is still surplus value accrued to your company (if you do good work for your company, you will be contributing to their reputation from which they will profit). However, if you are primarily writing code, the vast majority of your labor is producing a product that you are fundamentally not alienated from.

Proudhon, while sprinkling the word revolution throughout his writings, was notably not particularly revolutionary in his calls for action. He was reformist. Rather than calling for a direct confrontation with capitalism, he called for worker-run collectives to be formed throughout the world to make capitalism irrelevant. While many people point to corporate use of free software as an argument for why free software and capitalism are perfectly compatible, the truth may be more complicated. Fundamentally, and in a non-confrontational way, free software seems to undermine one of the basic tenants of capitalist: worker alienation.

Now we just need to work on all the other ways corporate work places are alienating…

Removing the user list from gdm

Sun, 08 Jan 2012 09:32:44 +0000

UPDATE

Nothing lasts for ever. I just restarted gdm3 (July 29, 2012) and my list of users again popped up.

None of the tips in my original blog or the ones suggested in the comments made any difference.

With help from dkg’s grepping … we found in /etc/gdm3/greeter.gsettings:

#disable-user-list=true

If you don’t have that setting, you can add it under the [org.gnome.login-screen] section. If you do have it, just un-comment it.

No need to dpkg-reconfigure, just restart gdm3.

Also, don’t be fooled by IncludeAll or Include in /etc/gdm3/daemon.conf. Those settings don’t seem to have any affect.

INFO BELOW IS OUTDATED

I don’t want to advertise the list of available logins when gdm starts and presents a login screen on my laptop.

Seems harder to figure out how to do this than it should…

Thanks to a helpful blog I’ve figured out two ways to accomplish the task:

sudo gconftool-2 --direct --config-source xml:readwrite:/etc/gconf/gconf.xml.mandatory --type Boolean --set /apps/gdm/simple-greeter/disable_user_list True

or edit:

/etc/gconf/gconf.xml.mandatory/%gconf-tree.xml

And add:

<?xml version="1.0"?>
<gconf>
  <dir name="apps">
    <dir name="gdm">
      <dir name="simple-greeter">
        <entry name="disable_user_list" mtime="1338752251" type="bool" value="true"/>
      </dir>
    </dir>
  </dir>
</gconf>

Servers4All... unless someone complains

Sun, 08 Jan 2012 09:32:44 +0000

On Wednesday, February 1, a new virtual server May First/People Link recently rented went offline. We contracted the virtual server through Server4All because we need their un-metered 100Mbit connection to help us handle the bandwidth for Sahara Reporters, one of the most important independent news sources for Saharan Africa. With the server offline, the web site was down as well.

We scrambled to setup alternative caching servers to handle the bandwidth.

When I logged into our control panel, I saw the message: This virtual server has been suspended by the administrator. Please contact support.

I immediately contacted support and then received the message:

Hello

We have received the following complaint associated with your server/service.

IP: 76.73.121.164

To prevent any further abuse we have suspended this service. In order to
resume, we request you to cooperate with our investigation as promptly as
possible. Please respond to us with the following details:

(1) What has caused the complaint
(2) What is the server used for. Purpose?
(3) How can you resolve the complaint and make sure it will not be repeated.

Depending on the nature of the complaint and your response, we will put back
the server online.  Please note, this has violated our Terms Of Service. We
expect your response within 24 hours, otherwise your account will be
terminated permanently.  Thank you

What?? What complaint?? I followed up but had to wait til the next day to get the response.

Here is the full log,

An email advertizing the Domain Name: saharareporters.com
has been sent to the blacklist.woody.ch spamtrap.

This Domain does resolve to IP addresses one of which your are responsible:
76.73.121.164

Please investigate why this Domain has been advertized.

Attached you find the headers and reports in ARF for automatic processing.
Feedback is appreciated.

Actual listing periods:

Bounce: 1 Hour in DNS.
Whitelisted IP: Not lised in DNS.
Spam: 24 hours in DNS.

Every Hit: 14 days in evidence DB.

For any questions or Feedback, contact abuse@woody.ch

From: is intentionally set to a bit-bucket.

Kind regards
-Benoit Panizzon-

There is no attachment. I went to woody.ch and it was in German. Then tried blacklist.woody.ch, but no luck. Finally I found the Woody’s World Blacklist Page. I plugged in our IP address into their checker and I got:

Output from the Check, if empty the IP is not listed.
164.121.73.76.[name of the blacklist] being tested.

Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)
Host 164.121.73.76.blacklist.woody.ch not found: 3(NXDOMAIN)

Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)
Host 164.121.73.76.rbl.maps.vix.com not found: 3(NXDOMAIN)

Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.mail-abuse.org not found: 3(NXDOMAIN)

;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached

;; connection timed out; no servers could be reached
;; connection timed out; no servers could be reached

Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.ordb.org not found: 3(NXDOMAIN)

Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)
Host 164.121.73.76.dev.null.dk not found: 3(NXDOMAIN)

Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)
Host 164.121.73.76.blackholes.five-ten-sg.com not found: 3(NXDOMAIN)

Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)
Host 164.121.73.76.bl.spamcop.net not found: 3(NXDOMAIN)

Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)
Host 164.121.73.76.relays.visi.com not found: 3(NXDOMAIN)

164.121.73.76.blacklist.spambag.org has address 208.91.197.182
164.121.73.76.blacklist.spambag.org descriptive text "v=spf1 -all"

So, Woody’s World thinks we are listed in spambag.org. I went to spambag.org and learned that the domain is for sale. I then tried blacklist.spambag.org and got the same page. This page has many links all pointing to advertisements. The “RBL List” link takes me to a page advertising “5 foods you must not eat.”

Next, out of curiousity, I tried Mxtoolbox. I got one hit from Barricuda. Barricuda says the reputation of the IP address is “poor”. Why? According to Barricuda, the reasons could be:

Your email server contains a virus and has been sending out spam.
Your email server may be improperly configured.
Your PC may be infected with a virus or botnet software program.
Someone in your organization may have an infected PC with a virus or botnet program.
You may be using a dynamic IP address which was previously used by a known spammer.
Your marketing department may be sending out bulk emails that do not comply with the CAN-SPAM Act.
You may have an insecure wireless network which is allowing unknown users to use your network to send spam.
In some rare cases, your recipient’s Barracuda Spam Firewall may be improperly configured.

Keep in mind, this IP address is not used for sending email. It’s just the web site.

I then took a step back and re-read the complaint and noticed that it says that the domain name saharareporters.com was listed in a spam email. Hm. More concerted searching for the terms “woody spamtrap blacklist” and I found a pattern in URLs that suggested I plug in the following:

http://news.scoutnet.org/rblhostlist.php?id=saharareporters.com.uri

And sure enough, there was a result. In short, it was a classic Nigerian Oil scam in which the person claims to be “JAMES IBORI ex-governor of DELTA STATE oil city.” The scammer acknowledges that he has been arrested, but promises lots of cash to the person who can help him. In an effort to boost their credibility, the scammer included a link to a Sahara Reporters article about the real James Ibori.

And that, my friends, is enough to have one of the most prominent independent African news organization taken offline.

~~But, Sahara Reporters shouldn’t necessarily feel singled out.~~ Sahara Reporters should feel singled out. They exist to illuminate news from Africa. They were taken offline because a series of individuals don’t know the difference between a prominent independent African news service and a criminal scammer. To most of the Internet, Nigeria and email scams are synonymous. That has to change.

Although Sahara Reporters is particularly vulnerable, any site hosted with Server4All can potentially be taken down. All you have to do is write a fake spam/scam email, including a link to the web site you want to be taken offline, and then send that email to: listme@blacklist.woody.ch.

I’m currently following up with both Woody’s World and Server4All. However, once this particular issue is resolved, we’re left with a much bigger and ominous problem. If your hosting provider (or their upstream provider) takes your site offline when it receives a complaint first and then asks questions second, you have a big problem.

All of our legal fights over our rights to keep content online are moot if our providers, without any legal pressure to do so, still take down our services based on spurious complaints.

Sparkleshare

Sun, 08 Jan 2012 09:32:44 +0000

Sparkleshare is a file sharing utility that keeps folders in sync on multiple computers in a similar fashion to the commercial and proprietary Dropbox. Sparkleshare recently released a Windows client, finally allowing me to start switching my co-workers at the Progressive Technology Project away from Dropbox.

Overall, I’m very impressed. In particular, I appreciate the Sparkleshare authors’ decision to build on top of existing tools (git for storage and revision control, ssh for transport, and ssh public key infrastructure for authorization and authentication). That means I don’t have to learn new tools and protocols to debug and it means Sparkleshare can focus on the file sharing pieces.

Despite my overall enthusiasm, I do have some serious concerns.

Protecting your credentials

A researcher found a startling security flaw in Dropbox - if you can copy a particular file from a user’s computer to your own computer, you can impersonate them flawlessly, accessing all their Dropbox files without having to know the user’s password. Unfortunately, Sparkleshare suffers from this same vulnerability if you use the default configuration.

When you first install Sparkleshare, it creates a password-less ssh private/public key pair and then makes the public part easily accessible to you so you can add it to your server (or gitorious account, etc). Very convenient. But it also means that all an attacker needs to do is copy your private key (and your Sparkleshare configuration file) and they get complete control over your files.

This problem is easily avoided. If you are running Linux or Mac OS X and you have your own key loaded in your ssh agent, Sparkleshare will happily use that key. So - simply by using your existing (presumably password-protected key) on your remote Sparkleshare servers, you can mitigate this problem. Sparkleshare will still load your Sparkleshare generated key, but if you don’t provide that key with access to anything, no harm is done.

Confirming each use of your key

However… that leads to a new problem. If you are like me, your ssh agent is configured to ask for a confirmation every time your key is used. And, Sparkleshare regularly polls the remote git repository for changes. At best repeatedly clicking to confirm is tedious. At worst, it prevents you from intelligently rejecting malicious requests, thus defeating the whole purpose of the check.

It is possible to launch Sparkleshare via ssh-agent in an environment in which you are not requiring the confirmation when Sparkleshare uses your key, but still requiring it for all other uses. However, given the trade offs, I’ve decided to add a password to my Sparkleshare provided ssh key rather than using my existing key:

ssh-keygen -p -f ~/.config/sparkleshare/sparkleshare.jamie@progressivetech.org.key

Now, I am prompted to enter my passphrase when I start Sparkleshare and don’t have to confirm every use of the key. And, I continue to confirm each use of my main key.

git was designed to store code, not documents

Just because something is designed for one purpose doesn’t mean it can’t be used for another. However, there are a few limitations.

Large files are one. git can handle files over 100MB, but may run into memory problems. I had to alter the git windowMemory setting, raising it higher than the size of the largest file.

Another problem is disk space. Since git keeps full revision history on every machine, you have to download more data than just the files that are checked out. The more edits you make to a repository, the more disk space beyond what is checked out is needed (and deleting files won’t help).

Lastly, you can’t use file modification times. With git, the file modification time will depend on when you checkout the files. With some fancy hook writing, you could tweak things so that the file modification date is the same as the commit date, but that still won’t help you if you add an existing directory to Sparkleshare because all the files will have the same commit time.

This “bug” has been reported to git and it has been rejected because mucking with the modification time of files can have bad results when you are using make to compile code. As Linus colorfully put it:

I'm sorry. If you don't see how it's WRONG to set a datestamp back to something that will make a simple "make" miscompile your source tree, 
I don't know what definition of "wrong" you are talking about.
It's WRONG.
It's STUPID.
And it's totally INFEASIBLE to implement.

Well, did I mention that git was designed for source code?

No server validation

Of all the problems I encountered, this one is by far the most serious.

Perusing ~/.config/sparkleshare/debug.log is very informative. On a default installation, after you have added a project, you’ll see:

19:32:27 [Fetcher][/home/jamie/SparkleShare/.tmp/bar] Fetching folder: ssh://foo@bar.org/foo/bar
19:32:27 [Fetcher] Disabled host key checking for bar.org

Wah.

Once the project is added, whatever key fingerprint is given is stuffed into your ~/.ssh/known_hosts file. Host key checking is only disabled when you initially create the project, so if you connect to the proper server the first time, you are protected from subsequent man-in-the-middle attacks. However… if the initial fingerprint is wrong, your personal ssh configuration is now poisoned. This seems like a very bad idea. I’ve opened a debian bug to address it.

Arriving in Dakar

Sat, 08 Jan 2011 09:32:44 +0000

We’ve safely arrived. Joseph, Ross and I have joined Mallory in Dakar for the World Social Forum. It’s the first time May First/People Link has had an opportunity to travel with four technologists.

Joseph and I met up with Ross in Washington…

Joseph and Ross before departing from Washington, DC

The flight was smooth, landing in Dakar at 6:00 am.

First Stop: Bakery

Mallory, once again, found an amazing space for us to stay, complete with Internet access and art work. This place is way more posh than our lovely techie house in Detroit.

Dakar Mona Lisa

And, after going some insane number of hours without sleep, we had dinner with several other members of the international.

International Team

More to come!

Beach Day

Sat, 08 Jan 2011 09:32:44 +0000

We took the afternoon off today. A 15 minute cab ride followed by a short 5 minute boat ride took us to N’gore Island.

Joseph on the Boat to N’gore Island

Abondoned structure on N’gore Island

Joseph, Ross and Mallory in the ocean off N’gore Island

Capitalism Is Crashed

Sat, 08 Jan 2011 09:32:44 +0000

The Occupy Wall Street protests taking place around the world are incredibly moving.

And, at May First/People Link, the joint us jumping with new projects. Occupy Together was the first to join, followed by the Occupy Wall Street Journal. We then launched boggs.mayfirst.org as our dedicated Occupy Wall Street shared server and the radical tech work keeps on coming (OccupyTechnoloyg, OccupyProvidence, OccupyColumbus GA just to name a few).

It’s also great to see free software and responsible software as a service making it’s way into the conversation. I’ve heard many reports about the politics of technology taking it’s rightful place in the realm of political decisions being made.

I don’t have the context for this photo from the blog non ti stavo cercando, but it sure sums up the connection :

Dakar Blog Workshop

Sat, 08 Jan 2011 09:32:44 +0000

May First/People Link collaborated with Mousa from Rwanda to teach our first workshop at the convergence center. Mousa handled the french speaking participants while Ross, Joseph and Mallory trained the english speakers. The resulting web site brought tears to my eyes.

Data Security

Sat, 08 Jan 2011 09:32:44 +0000

The way we evaluate Internet technology is changing … quickly.

Seems like you can’t read the news these days without hearing about a security breach leaking hundreds of thousands of email addresses, a “redundant” cloud service being down for over two days, or a mobile device maker accidentally tracking your movements.

There was a time when most movement activists picked technology based on functionality and intuitive and well-designed user interfaces.

However, now that we’re more dependent on more technology for things more important to our lives, we’re finding that we need to add a deeper foundational integrity to our list of criteria.

Sadly, as May First/People Link members learned earlier this month, data breaches don’t just happen to profiteering corporations looking to cut corners to make a quick buck. In the worst data loss of our organization’s history, we were unable to restore databases for about 20 of our members after a server had to be re-build from the backup. The failure, which was a calamity for the members involved, was the result of a series of mistakes on our part.

While we were not able to bring back the data, we did our best to model an honest and productive response, starting with a straight forward report of the problem, personal contact to all affected members (we missed a few due to in-correct contact information), a member-wide email invitation to a meeting to address the situation, a set of proposals to ensure it won’t happen again, and an ad-hoc committee to see those proposals through.

The importance of our data and software will only grow in the future. There’s no question we will need to continuously react to breaches affecting us, whether it’s on May First/People Link or any one of the corporate providers our movement is still dependent on. Let’s also remember to be pro-active - to conciously discuss and act to build a network we can trust.

IMC Convergence Center Day One

Sat, 08 Jan 2011 09:32:44 +0000

Fabian, from London and Molefi from South African, arrived early monday morning - and were instrumental in our last day preparations for the IMC Africa convergence Center.

Advance Team pictured on the 2nd floor terrace of the House

Turns out we got the numbers wrong - 30 delegates arrived on the day we negotiated access to the first two bedroom apartment. Yikes. Fortunately, the house was ready a day early and our landlord was willing to let us move in a day early.

The “mattresses” are one inch foam pads, but everyone is very flexible and in good spirits.

On day 1 (Wednesday), we met in the morning and formed four committees: finance, program, welfare, and mediation.

Preparing our first lunch

After the committees met, we had down time while we waited for lunch.

In the backyard

Privilege Separation

Sat, 08 Jan 2011 09:32:44 +0000

What are the biggest security threats to my laptop? Almost all the software I install is vetted and signed by a member of the Debian team. I run reliable virus software on our mail server. My disk is encrypted and xscreensaver locks my screen after a few minutes of inactivity. What else?

The two biggest threats I’ve recently considered are: web browsing and non-free software or software that doesn’t come from Debian (I regularly have to use skype and zimbra, for example).

To mitigate these risks, I’ve configured these programs to run as their own users, thus adding a layer of separation between the programs and my primary user account, which has access to my gpg/ssh keys and email.

With a program like zimbra it’s fairly easy. I created a zimbra user, added my primary account’s public ssh key to /home/zimbra/.ssh/authorized_keys, and add the following stanza to ~/.ssh/config:

Host zimbra 
Hostname localhost 
User zimbra 
ForwardX11 yes

Now I can start zimbra with:

ssh zimbra ./zdesktop-launch

Skype was a little harder, since the skype client has to access the audio system. With pulseaudio, though, it’s a snap. I copied /etc/pulse/default.pa to ~/.pulse/ and added the line:

load-module module-native-protocol-tcp auth-ip-acl=127.0.0.1

Then, I added /home/skype/.pulse/client.conf with the contents:

default-server = 127.0.0.1

[Note: Jamie Rollins and dkg have pointed out that this arrangement allows any user on my laptop to send arbitrary data to pulseaudio, running as my primay account. They suggested configuring pulseaudio to listen on a unix-domain socket, and then configuring pulseaudio to only permit access to users in a particular group.]

iceweasel is the most complicated. In addition to the pulseaudio trick, I had to make two other allowances.

First, there are a lot of processes that launch a web browser in a number of different ways (sometimes asking for a new session, other times adding a tab to an existing one, sometimes passing an URL as an argument, etc).

The one that I got the most stuck on was mutt. Sometimes I want to see how an HTML message looks in iceweasel. Via mailcap, mutt creates a temp file with the html content and then launches a web browser to view the file. As can be reasonably expected, this temp file is owned by my primary user, and only read-able by the owner. That means my iceweasel user can’t read it.

Eventually, I decided the easiest way to deal with these various scenarios was to write a simple bash script to launch my web browser (see below). I registered it via update-alternatives, so most reasonable programs that want to launch a web browser will use it.

The second issue is that I use the monkeysphere xul plugin to verify TLS certificates, which requires iceweasel to communicate with my monkeysphere validation agent. My agent runs as my primary user and by default only responds to queries from my primary user.

Fortunately, monkeysphere is well-designed and can handle this situation. As you can see from my web launcher script, I pass MONKEYSPHERE_VALIDATION_AGENT_SOCKET=$MONKEYSPHERE_VALIDATION_AGENT_SOCKET when calling iceweasel. In addition, I added the following before I exec monkeysphere-validation-agent:

export MSVA_ALLOWED_USERS="iceweasel jamie"

With this simple infrastructure setup, it’s possible to easily isolate future programs as well.

Lastly… here’s the script for launching iceweasel.

#!/bin/bash

stdin=
new_session=no
url=

for arg in "$@"; do
  if [[ "$arg" =~ ^-- ]]; then 
    if [ "$arg" = "--new-session" ]; then
      new_session=yes
    elif [ "$arg" = "--from-stdin" ]; then
      stdin=yes
    fi
  else
    url="$1"
  fi
done

if [ "$stdin" = "yes" ]; then
  temp=$(mktemp)
  while read line; do
    echo "$line" >> "$temp"
  done
  # it must be readable by the iceweasel user
  chmod 755 "$temp"
  url="file:///$temp"
fi

args=
if [ "$new_session" = "yes" ]; then
  args="--no-remote -ProfileManager"
fi

if [ -n "$url" ]; then
  args="$args '$url'"
fi

ssh iceweasel "MONKEYSPHERE_VALIDATION_AGENT_SOCKET=$MONKEYSPHERE_VALIDATION_AGENT_SOCKET iceweasel $args" &

[ -f "$temp" ] && sleep 5 && rm "$temp"

Comments by email…

Hi,

http://current.workingdirectory.net/posts/2011/privilege-separation/ has good intentions but afaik it does not improve security much. X applications can sniff your passwords and inject commands to your terminal emulators.

I personally use xpra to get a similar solution without the hazards of X. I’ve been using it for two months now both at work and home. There are still bugs but svn version is getting better all the time.

See for example my reply to

https://grepular.com/Protecting_a_Laptop_from_Simple_and_Sophisticated_Attacks

Timo

Puppet Without Masters

Sat, 08 Jan 2011 09:32:44 +0000

Puppet labs’s use of the term puppetmaster is rather clever (in contrast to other un-necessarily offensive uses of “master” in the software world).

While I appreciate the clever name, I’m less impressed with the concept.

At May First/People Link we’ve spent the last several years (including the last couple months in earnest) working to transition management of our 90-some servers from a collection of hand-written bash scripts to puppet.

Over the years, we’ve worked hard to keep our servers as secure as possible. We have a team of about a half dozen people who all have root access on all servers. It’s all key-based access. To help mitigate a disaster if one person’s keys were compromised, we’ve implemented monkeysphere on all servers, allowing us to easily revoke access.

After spending so much time thinking through our root-access strategy and fully implementing the monkeysphere to reduce our exposure to a single point of vulnerability, I was disappointed by puppet’s use of a puppet master. For those less familiar with puppet, it goes something like this:

One server (or god forbid multiple servers), run an externally accessible daemon. Each and every server on your network runs a daemon as root that periodically communicates with the puppet master, receives new instructions, and then (again, as root) executes these instructions.

In other words, if your puppet master is compromised, I’m not sure exactly what you would need to do, short of rebuilding every server in your network.

To make matters worse, it seems as though some users generate and store all server ssh keys (private and public) on the puppet master and then push the private keys to their respective nodes. That means an intruder doesn’t need to write to the puppet master, just reading these keys would be enough to compromise all servers in your network.

There must be a better way.

Puppet without masters

After some web-searching, I found a promising thread on the puppet list asking what’s lost without a puppet master. This thread lead to a couple other blogs by people who have worked out a system for using puppet without a master.

It turns out that there are two distinct points of centralization with puppet. One is the puppet master (as described above). In addition, there is a concept called storeconfigs - which allows each node in the network to store information in a central database. For example, one server can store a request for an account to be setup on a backup server. The next time the backup server runs, it checks the storeconfigs, finds the request, and creates the user.

It’s possible to run puppet with storeconfigs but without running a puppet master (that avoids the hassle of running the puppet daemons, while providing the convenience of centralization). For our purposes, however, we decided to forego both the puppet master and storeconfigs. We did not want any form of centralization that would provide an additional point of vulnerability.

As is common with puppet, we are storing our puppet recipes in a git repository. And, we are publishing to a single, canonical git repository on the Internet. On each node, we have two git repositories - one is a bare repo (that we can push to) and the other is a checked out repo (in /etc/puppet) that is read by puppet. The bare repo has a post-update hook the changes into the /etc/puppet directory, pulls in the changes from the bare repository, and runs puppet against the newly checked out files. Therefore, we can apply new puppet recipes to any server on the network with

git push <server>

No daemons: neither a master daemon nor a puppet daemon running on the node using up memory or providing a potential security hole. The git push happens over an ssh connection - since all system administrators already have root-level ssh access on every server - there is no need to grant any additional access above what we already have.

Pushing works great - but with 90 nodes we don’t want to have to push to 90 servers everytime we want a change made. That’s where the canonical git repository comes in. A cron job runs a script on each node once an hour that runs git remote update from /etc/puppet. The script then checks the time stamp on the most recent gpg-signed tag and compares it with the time stamp of the current commit. If the most recent gpg-signed tag is newer, it verifies that the tag came from a list of authorized gpg keys (the very same gpg keys used by the monkeysphere to grant root level ssh access). If the gpg signature of the tag can be properly verified, then the changes are merged and puppet is run on the new recipes.

What about privacy?

One of the benefits of a puppet master setup is that nodes get configuration details on a need-to-know basis. The puppet master doesn’t share the entire puppet repo - only the compiled manifest for the node with which it’s communicating.

Our solution to this problem was to go screaming in the other direction. As you might notice from our support wiki and ticket system, we generally favor transparency. Since we are publishing our entire puppet git repo publicly, there seems little point in trying to hide one node’s configuration details from another node.

That also means each node carries around about 4Mb of extra weight in the form of disk space for the git repo. That seems like a small price to pay for the resource savings of not running a puppetd process all the time.

More differences

As I’ve read the puppet lists, faqs and documentation, I’ve found yet more ways our use of puppet diverges from the norm.

The first is a little thing really - most people seem to store all their node configurations in a single nodes.pp file. I’m not sure why. Fortunately, puppet’s include syntax allows globbing, so we’ve created a directory and gave each server it’s own .pp file. This arrangement makes it much easier to parse the configuration with tools other than puppet (like, Q. How many servers do we have? A. ls | wc -l).

Backup and Nagios monitoring without storeconfigs

More significantly - there are some things we can’t do since we are not using storeconfigs. While many puppet users add a variable, like $nagios = true before including their sshd class (which then causes the sshd class to store a configuration for the nagios server to monitor ssh on the node in question), we were forced to come up with alternatives.

My first solution was to simply list all the servers that needed to be monitored in the server node configuration file for the nagios server. Ditto for the backup servers. This approach, however, proved cumbersome and error prone. When adding a new node, I now have to edit three files instead of one. And, how can I easily tell if all nodes have their nagios and/or backup configurations set? The solution was rather simple - there’s more than one way to store a config for another node. Our nagios server is called jojobe.mayfirst.org and our backup server is luisa.mayfirst.org. A typical node.pp file looks like this:

node pietri.mayfirst.org {
  # node config goes here

}
if ( $fqdn == "jojobe.mayfirst.org ) {
  nagios_montior { "pietri": }
}
if ( $fqdn == "luisa.mayfirst.org ) {
  backup_access { "pietri": }
}

This way all configuration related to pietri stays in a single file.

Host keys and granting access between servers

storeconfigs is commonly used to distribute host ssh keys. Every node that is added to puppet has it’s ssh host key stored centrally and then re-distributed to every other node. That way, you can ssh from node to node without ever getting the ssh fingerprint verification. Avoiding that prompt is particularly important when backing up from one server to another via automated scripts. storeconfigs can additionally be used to copy user’s public ssh keys - thus granting user access between servers.

Our solution to this problem: monkeysphere. Rather than maintaining our own private data store of keys, we publish (and sign) our ssh keys via the web of trust. In addition to server keys, each one of our servers’ root user has an ssh-enabled gpg key (also publicly signed by us). By configuring each server to trust our system administrators’ gpg keys for verifying other keys, we can both avoid the ssh fingerprint manual verification step and we can grant a root user on one server access to another server by simply dropping root@$server.mayfirst.org into an authorized_user_ids file on the target server.

There’s no question - the setup was rather tedious (we’re using runit to maintain an ssh-agent for each root user), however, now that’s in place (and configured via puppet), it’s a breeze to add new servers. The only extra step we have to take is to confirm and sign each new server’s keys. This “extra” step not only allows our servers to verify each other, but also allows our users to verify the servers as well, so it’s hardly an extra step at all.

Shared modules

There’s a vibrant community of third party module developers for puppet. Rather than figure out the intricacies of having puppet configure sshd, for example, you can install a contributed sshd module and then you simply add:

include sshd

And you get a default sshd setup. Many of these modules are fairly well developed, offering the ability to easily customize your setup in a number of different ways.

Unfortunately, most of the modules assume you are using storeconfigs and if you are not, they will either fail to work right or you will get noisy errors. At first, this seemed like a problem. However, as I built our puppet recipes, I found myself increasingly frustrated with the third party modules that we could use.

Configuring servers is hard - and requires constant debugging and trouble shooting. puppet already provides a layer of abstraction between you and the server you are setting up. Given the benefits of puppet, I’m willing to spend the time learning the puppet syntax and asking the rest of our system administrators to do the same. This layer of abstraction is further compounded by our use of git to store the configurations (not a problem if you are git hero - but most of us are already struggling to get a handle on using git). Again, all seems worth it for the pay off.

Now enter the puppet module. In addition to learning puppet syntax (and struggling with git) you now need to understand how the third party module works. With software programming, I typically don’t need or want to learn how a library or class does what it does - that’s the beauty of object-oriented programming: it hides the complexity. But when it comes to configuring the servers that I will be responsible for debugging and maintaining, I really need to know exactly what is happening.

To further compound the problem, I found myself wading through third party module code designed to work on Debian, Ubuntu, CentOS, Redhat, gentoo… and more. We run entirely on Debian - we don’t need any of this extra code. And, once I got rid of all the other operating systems, I was still left with a complex module that allows you to configure software in ways we’ll never need.

In the end, we tore out most of these third party modules and replaced them with file and exec puppet resources that did exactly what we needed them to do. Our code base is now much smaller and simpler.

Not just a whiner

I have a lot more to whine about (like why native resources for things like nagios that are so easily handled with the file resource?).

However - the remarkable thing about puppet is that it’s flexible. Despite some fairly substantial problems with the “typical” use of puppet, the program provides enough flexibility for us to use it in a way that fully meets our needs. After having built my own bash-based set of configuration scripts and deeply exploring puppet, I have a great appreciation for the difficulty of building system configuration software (we considered and rejected cf-engine and chef as not being any better).

And, if you are still not convinced that puppet will work fo you … you might consider a package I learned about after going down the puppet route: slack.

Slow Drip and other tales from Dakar

Sat, 08 Jan 2011 09:32:44 +0000

We’re slowly adjusting to daily power outages. We’ve only had one that last 24 hours… most are just 1 - 2 hours.

Joseph earns the nickname Slow Drip making coffee in the dark kitchen

Senegal is not exactly Tunisia or Egypt, but the increasingly frequent and prolonged power outages have caused some tire-burning in the streets.

Fire in the streets

If you are curious about food…

Breakfast in Dakar

I took a break from working to pose for this picture. The writing says: Le Culte du Travail (The cult of work). We found this in the University.

Le Culte du Travail

In other news, the convergence center house has been rented. It’s a great house

huge, 2 floors, 5 bedrooms and, somehow, 5 bathrooms. Joseph and I just walked the 15 minutses to the University (another 15 minutes to the library where the media center is). Wow. I’ve never attended a social forum within walking distance to where I was staying! We will be fitting about 30 of us into the house (and the two apartments down the streets). We will move into the house on February 2.

Testing x509 Certificates

Sat, 08 Jan 2011 09:32:44 +0000

Navigating the openssl suite of subcommands is time consuming.

Here’s my list of frequently used commands for easy reference:

Examining the certificates being offered by a host (replace imap/https with any service, -starttls imap can be eliminated if you are checking a service that provides tls from the start):

openssl s_client -starttls imap -showcerts -connect chavez.mayfirst.org:imap
openssl s_client -showcerts -connect chavez.mayfirst.org:https

Examine a certificate signing request:

openssl req -text -verify -noout -in /path/to/file.csr

Examine a x509 certificate:

openssl x509  -noout -text -purpose -in path/to/file.crt

We need better metaphors

Sat, 08 Jan 2011 09:32:44 +0000

Most of us developers use metaphors to help convey an understanding of how our software works. Sadly, many developers choose the “master” and “slave” metaphor. Why would we voluntarily choose such an ugly human human interaction with a real and present history in many of our lives to describe software that we’ve written?

I wouldn’t rule out the possibility that someone will (or already has) invented software for which the master/slave metaphor aptly conveys its function, however, I’d prefer not to use it. Also, many people intentionally play with master/slave relationships because of the history and connotations it provokes.

However, use of this metaphor in software like MySQL databases or Bind Name servers seems pointless, offensive and mis-leading.

I’ve never seen the use of the master/slave metaphor in software that conveyed more than the idea that one piece of the software essentially controls, wholly or in part, another piece of software. So, why master/slave? There are many metaphors to choose from - parent/infant, boss/worker, guard/prisoner, king/subject, landowner/serf, top/bottom, etc.

What does master/slave convey that the others don’t convey?

Or, maybe they all convey the same thing, so it doesn’t make a difference which one you choose (after all, it’s just a metaphor). In that case, how about replacing master/slave with the metaphor husband/wife? Whoops. Now that’s offensive (if you don’t think that’s offensive you can stop reading here and please don’t post any comments - I won’t change your mind).

But, that’s different right? At one point in history, the role of the wife was to do as her husband says, but it’s 2011, and we’ve evolved beyond that, so by using husband/wife we’d be re-enforcing that ugly, sexist stereotype.

However, how is that different from master/slave? It is 2011 after all. Isn’t the role of the master to abdicate and the role of the slave to revolt?

Maybe that’s why Bind has so many security problems…

A Response to Mark Engler's 'The Limits of Internet Organizing'

Fri, 08 Jan 2010 09:32:44 +0000

Mark Engler’s customary insightful analysis is glaringly absent from his recent piece on Internet organizing in Dissent Magazine.

Engler frames his piece with:

I will state up front the conclusion that almost all articles of this sort come to: the Internet is a tool. It is potentially a rather useful tool, but it is not more than that, and it is no substitute for person-to-person organizing.

This conclusion is the correct one. Still, it never seems to quell the high-tech evangelists or to sink in with the public at large.

Forget the high-tech evangelists - let’s stick with the public at large. Why doesn’t it sink in with us?

Engler devotes one paragraph to this question:

Partially, I think this is a product of the widespread failure, outside of social movement circles, to understand what organizing really is. You can raise money on-line, you can widely disseminate information, and you can get people to sign a petition. But it’s very difficult and rare to be able to use the Internet to build the types of deep relationships that move people to make serious commitments to and sacrifices for social movements.

Inside social movement circles, we’ve perfected our ability to criticize each other for not being “real” organizers. This debate is as tiring as it is unproductive. What if we broadly define organizing as affecting what people do and how they interact with each other? From this perspective, technology has historically had an enormous impact on how people organize and are organized. The automobile, mass transit, television, and many other new technologies have had a fundamental impact on our organization. The Internet, in terms of its impact on society and how we organize, is no different.

Perhaps that’s why the “public at large” has such a hard time believing that the Internet is just another tool - we have experienced the huge impact the Internet has had on our lives. Unlike the “best organizers” Engler knows, who “hardly have time to check their e-mail,” most of us are not afforded that luxury. Checking our email, something unheard of 20 years ago, is not something we do when we simply feel like it. It’s a necessity. It’s the way we do our jobs. It’s how we are staying in touch with our friends and family.

And, unlike the other major technology changes of the 20th century, the Internet is based in democratic communication, affording us the ability to exchange ideas with any one of the Internet’s nearly 2 billion users. That capability is intoxicating for the public at large, intoxicating enough that we suffer through incredible amounts of frustration and often pain learning something new and dealing with difficult new protocols, both technical and social.

This mass movement of the world’s population to the Internet, in a quest to communicate and connect with others, hasn’t been missed by the corporate class. Facebook, Twitter and others, are investing millions of dollars in an effort to profit from this movement, control the Internet so it can more predictably produce profits, and restrain it’s open and democratic nature.

Engel’s piece not only fails to address this dynamic, but he dangerously equates the Internet with these corporations, depoliticizing the Internet as if it were just a hammer. Unlike a tool, the Internet is a fast-developing universe of political struggle. We are engaged in constantly shifting battles over whether or not the software that will increasingly control our lives should be free and open or owned by a for-profit company; who should control and have access to our data; whether our communications should be accessible to our governments; whether our ability to communicate is a right or a privilege.

Engel’s take on these struggles seems to be: “Don’t worry about it, go back to the phone bank.”

How the left should use the tools of the Internet is a fine discussion - one the left is and should continue to have. However, more critical to the future of our movement, is: what is our role in the development of Internet? How do we support and develop the revolutionary potential in the Internet? How will the Internet shape our lives in the future? If we choose to fully engage in this debate, put our resources behind our convictions and principles, we will have a very different future.

Benchmarking PHP

Fri, 08 Jan 2010 09:32:44 +0000

What’s the best way to run PHP in Debian in a shared environment?

Years ago May First/People Link switched from the old reliable mod_php to suPHP because mod_php ran all php scripts as the same user, meaning (among other things) that any site could access the database configurations of any other site.

suPHP was great, however, after several years we’ve decided to replace it because:

As our members increasingly use Drupal and Drupal sites grow increasingly more complex, suPHP’s performance has become a problem.
suPHP executes with the permissions of the owner of the file it is executing. From a zero-conf perspective, this features makes suPHP very easy to setup. However, from a security perspective, it forces you to run your php script with the permission to delete itself. Not optimal.

We first turned to fcgid. We tried it out on a couple individual web sites (fcgid can run alongside suPHP), and, after a week without much problems, we implemented it on an entire server.

Out of the box, on a shared server with several dozen web sites, the most notable resource utilization difference between fcgid and suPHP is the increase in memory usage. Unlike suPHP which dies after each requests, fcgid gains a performance boost by sticking around (with either 2 or 3 processes living on) to server future requests. If you have several dozen web sites that aren’t high traffic sites, it means they may each have up to the three processes staking out memory that would otherwise be free with suPHP.

We addressed the memory problem by adding the following to /etc/apache2/mods-available/fcgid.conf:

DefaultMinClassProcessCount 0
IdleTimeout 60
IdleScanInterval 60

These directions tell fcgid to kill processes that have been idle for more than 60 seconds. That means we still get the performance gain from a site getting more than one page view per minute, but otherwise we re-claim the memory.

Before switching our remaining servers we decided to do more extensive testing to ensure we would be getting the performance and resource gains we were hoping. Also, we decided to consider mpm-itk, which is a variation of mpm-prefork that runs each virtual host with a per-vhost configured user and group (like suexec, but for the entire virtual host). Since it’s a variation of mpm-prefork, it’s safe to run it with the more resource friendly mod_php.

We used the apache benchmarking tool ab. And we tested on a virtual machine (running on my laptop) with 1GB of RAM. The test web site we used was a copy of the May First/People Link site (we used the home page for the test), which is running Drupal 6. It’s a relatively simple Drupal site, however, we turned Drupal caching off to better simulate PHP processing usage likely to be found on our members’ sites.

We ran tests against the following configurations:

Plain mpm-prefork with mod_php and apc (as a standard to strive for)
Both mpm-worker with suPHP and mpm-prefork with suPHP (how much of a difference does worker vs prefork make with a Drupal site?)
mpm-worker/fcgid/suexec
mpm-itk with mod_php

The tests were run with:

1 request (in the results, this test is labelled 1:1)
20 requests, 5 concurrent (20:5)
50 requests, 10 concurrent (50:10)
70 requests, 10 concurrent (70:10)

We originally tried, for the last test, 100 requests/25 concurrent but it caused ab to time out on the mpm-worker/fcgid/suexec configuration. We then lowered it to 75/15. Still timed out (load on our test server went way up).

The tests paused 120 seconds between testing each environment to give the server load a chance to settle down. And we paused 30 seconds between each test (not enough time for the fcgid processes to be killed, but enough time for all the processes to hopefully complete their requests).

The tests are scripted and the full results are available. You can also see the various vhost configurations (test.mouse, test.mouse.itk, test.mouse.suexec.fcgid), and the fcgid.conf.

To summarize the report … if you are running one web site on a server, there’s little argument for using anything other than mod_php and mpm-prefork - the performance of this combination far exceeds any other combination. I’m scratching my head over the the vastly different conclusions reached by 2bits. They didn’t publish their tests so I’m not sure exactly how to compare. Maybe they have a lot more static content? Or - maybe there’s a mistake in our tests? Open to ideas.

suPHP seems to be left in the dust compared with either fcgid/suexec or mpm-itk. Given that it’s also less flexible, there’s not much going for it.

And lastly… fcgid/suexec and mpm-itk seem comparable. That was a surprise. We were expecting mpm-itk to perform better because it’s running mod_php. Given the similarity in performance, fcgid/mpm-worker seems like a much more well-tested and well-supported approach for a shared server.

One side note… originally, out of laziness, we tested against the default Drupal install.php page. Less static content and much less PHP processing. In these tests, with low loads, the results provide different numbers, but the comparisons between environment were similar … except with the last test. With the 70 requests/10 concurrent test, fcgid/suexec dramatically out-performed everyone (by a factor or more than 2). That’s probably because no new cgi processes had to be spawned and the effect of mod_php using APC was reduced because there was very little php code to process.

California Holidays 2010

Fri, 08 Jan 2010 09:32:44 +0000

The trip started out great. Taking a walk near the house…

… Meredith in the backyard

We even took a walk on the beach

Taking our chances with the tide.

Breaking the law (the beach we walked on wasn’t exactly open)

After our flight was rescheduled for wednesday, we thought we’d rent a car. We were never going to get this one.

After our flight took off from LA and then turned around and landed in LA, we decided to go with Amtrak.

We spent a lot of time looking out the window, particularly in the south west.

We splurged for the LA - Chicago leg - an actual sleeper car.

Six hour layover in Chicago. If you wait long enough you can find an open bathroom in the fanciest of places.

You can also see expensive public art…

… and inexpensive public art…

COP16 is coming soon

Fri, 08 Jan 2010 09:32:44 +0000

COP16, the meeting of the United Nations Framework Convention on Climate Change and the Kyoto Protocol, is coming up in early December.

The events taking place during these two weeks in Cancun, Mexico are mind boggling. There are at least two full scale paralell forums (Kimaforum and Diálogo Climático), caravans are arriving from around the world, and La Via Campesina is calling for 1000 Cancuns to take place around the world.

May First/People Link is working with our member Grassroots Global Justice and the Grassroots Solutions for Climate Justice - North America to build out a web site to help bring together as much of the information about the events as possible.

Stay tuned for the launch, coming soon!

From vservers to KVM

Fri, 08 Jan 2010 09:32:44 +0000

Given the impending deprecation of vservers, I’ve decided to make the switch to KVM on my laptop. Although lxc is a closer approximation to vservers, I decided to go with KVM due to it’s support in Virtual Machine Manager.

My first step was to confirm that my CPU would support kvm:

egrep -o "svm|vmx" /proc/cpuinfo

If that command outputs either svm or vmx (depending on whether you have Intel or AMD hardware) then your CPU supports virtualization.

I’m working on a host machine called chicken, which has a logical volume called vg_chicken0. All vservers on chicken operate on a root filesystem that is backed by their own logical volume.

In this post, I’ll describe the steps to convert the vserver hobo (which operates on a filesystem mounted on the host in /var/lib/vservers/hobo and is backed by the logical volume called vg_chicken0-hobo_root).

Both chicken and hobo are running debian squeeze.

vservers don’t have a kernel installed or grub. KVM virtual servers need both.

I was hoping I could simply enter the vserver, install both a kernal and grub and be ready to go. However, grub installation will fail miserably because grub can’t figure out how to install on the underlying disk (which is hidden from the vserver).

Next, I tried launching a kvm instance, passing a debirf generated ISO with the -c (cdrom) option. However, grub recognized that it was being installed onto a device that did not have a partition table (the logical volume was directly formatted with a file system).

So, since I had disk space to spare, I created a new logical volume:

lvcreate --size 5GB --name hobo_root_new vg_chicken0

I then added a gpt partition table (why not prepare for the coming 2TB disks?) and created two partitions. One partition for grub2 and one for everything else:

parted /dev/mapper/vg_chicken0-hobo_root_new mklabel gpt
parted /dev/mapper/vg_chicken0-hobo_root_new unit s mkpart biosboot 2048 4095 
parted /dev/mapper/vg_chicken0-hobo_root_new set 1 bios_grub on 
parted /dev/mapper/vg_chicken0-hobo_root_new unit s mkpart primary 4096

When prompted for the end of the last partition, choose: -1 and accept the adjustment.

I had to eyeball cat /proc/partitions to figure out which dm device was the second partition (dm-19).

I then created a file system:

mkfs -t ext3 /dev/dm-19

Mounted it:

mount /dev/dm-19 /mnt

And rsync’ed the data:

rsync -a /var/lib/vservers/hobo/ /mnt/

With the data in place, I chroot’ed and installed the packages I needed. When prompted, I chose not to install grub to the disk, because I wanted to wait until I had an environment in which the proper disk would be available to grub as it will when the virtual server boots (see below):

chroot /mnt
mount /proc
aptitude install linux-image-2.6-amd64 grub2
umount /proc
exit

Then, I cleaned up:

umount /mnt
umount /var/lib/vservers/hobo
lvremove vg_chicken0/hobo_root
lvrename vg_chicken0/hobo_root_new hobo_root
dmsetup remove /dev/mapper/vg_chicken0-hobo_root_newp1
dmsetup remove /dev/mapper/vg_chicken0-hobo_root_newp2
kpartx -d /dev/mapper/vg_chicken0-hobo_root

And I removed it from /etc/fstab.

Next, I created a new kvm virtual server, using the disk /dev/mapper/vg_chicken0-hobo_root and passing a debirf cd image with -c:

virt-install –name hobo –ram 512 –disk /dev/mapper/vg_chicken0-hobo_root -c /usr/local/share/debian/ISOs/debirf-rescue_squeeze_2.6.32-5-vserver-amd64.iso

After logging in, I installed grub2 (aptitude update; aptitude install grub2) and then I installed grub:

mount /dev/sda2 /mnt/
grub-install --no-floppy --root-directory=/mnt /dev/sda

After running grub-install, edit /mnt/boot/grub/device.map so it reads:

(hd0) /dev/sda

Then, rerun grub-install command.

I tried generating the grub.cfg file, but got an error message indicating that grub-probe would not detect the device providing / (because I was running on a ram file system from debirf).

I added the following to /mnt/etc/fstab:

/dev/sda2  /     ext3 errors=remount-ro 0 1
proc       /proc proc defaults          0 0

And then re-generate the initrd image:

chroot /mnt
mount /proc
update-initramfs -u

So, I rebooted the virtual machine by typing:

exit
reboot

This dropped me into a grub shell. I manually typed:

root (hd0,gpt2)
linux /vmlinuz root=/dev/sda2 ro
initrd /initrd.img
boot

Once booted, I logged in a completed the task with:

update-grub

Growing disks: upgrading from 1 TB to 2 TB disks

Fri, 08 Jan 2010 09:32:44 +0000

We’ve got a fairly layered set of software that runs beneath our file systems: RAID1 followed by cryptsetup, followed by lvm. It works remarkably well.

However, when it’s time to do something like enlarge a partition or disk, it gets complicated.

We have a server (hubert) that was running two 1 TB disks in a RAID1. We typically create two partitions: one small boot partition and one large partition the uses the rest of the available disk (with 2 TB disks we’ve added a third partition for grub).

Last month I replaced one of the disks with a 2 TB disk, partitioned the new disk with similar a partition scheme (except that the “rest of the disk” partition was twice the size as the original one), and then added the new partitions to the RAID (mdadm /dev/md0 –add /dev/sda1; mdadm /dev/md1 –add /dev/sda2).

Yesterday, I replaced the 2nd 2 TB disks, and again added the new partitions.

The result was a fairly painless transition from 1 to 2 TB disks with minimal downtime.

The problem - though - is that we’re not using all the disk space available, since md1, the RAID array that uses the “rest of the disk” partition, is only about 1 TB when it should be just under 2 TB.

That problem was easy enough to solve by growing the RAID device with:

mdadm -G /dev/md1 -z max

It took a while, but eventually, running:

cat /proc/mdstat

And

cat /proc/partitions

Reported the right sizes for md1.

After RAID, comes cryptsetup (/dev/md1 is the source device for md1_crypt). So, the next command was:

cryptsetup resize md1_crypt

In the output of:

cat /proc/partitions

I saw the change reflected in the dm-0 device.

Since md1_crypt is used as the physical volume for a volume group, the next step was to resize the physical volume:

pvresize /dev/dm-0

This change was reflected with:

pvs

Now, the command:

vgs

Displayed a full 931 GB available that wasn’t there before.

Next up: resizing the logical volume:

lvresize --size 1700GB vg_hubert0/blanco_members

And, last but not least, was the file system:

 resize2fs /dev/mapper/vg_hubert0-blanco_members

See also dkg’s article on resizing crypt partitions.

May First/People Link participates in 1000 Cancuns

Fri, 08 Jan 2010 09:32:44 +0000

We’ve setup a video watching station at Blue Stockings Bookstore, where we’re handing out print outs of the excellent climate solutions briefs on grassrootsclimatesolutions.net.

We’re also preparing for the protest in fron the United Nations scheduled for 5:30 pm tonight. I just finished a slide show that will be projected.

Net Neutrality: May First/People Link Takes Two

Fri, 08 Jan 2010 09:32:44 +0000

When net neutraliy first surfaced as an issue years ago, it was hard to get behind. Not the principle - which states that providers of Internet access should supply a connection without restrictions or preferences based on where or who the content comes from - that should go without saying.

Instead, it was hard to get behind political organizing in support of this principle. At the time, the debate was defined by two corporate giants: Google was standing in support of net neutraliy, and Verizon was angling to charge content providers like Google money to be given preferential bandwidth treatment.

The battle grounds became the Google/Verizon corporate negotiating table and the US federal regulatory agencies - both historically have served as empowerment sink holes for anyone trying to build a political movement.

Furthermore, given Google’s various moves to control the Internet, it was hard to publicly support an issue so clearly staked out by them. With all the revolutionary work being done to develop free and open source software and protocols, the global convergences of the world social forums and climate change activists, stopping to help Google defeat an obviously bad idea proposed by Verizon seemed like a red herring.

As co-directors of May First/People Link, our leadership committee has given Alfredo and I the political responsibility to provide day-to-day leadership based on a solid assessment of every situation affecting the work of our organization and members. We chose to steer our organization clear of net neutrality, and now we realize that was a political mistake.

Last week, Google and Verizon announced that their feuding is over. They released a joint proposal for a legislative framework. Despite predictable applause from the tech mainstream, the implications are fairly profound and dangerous.

You cannot get away with standing up in the United States and saying that net neutrality is a bad idea. That’s like saying publicly that profit should come before people. Instead, the recommendation supports the principle of net neutrality when it comes to a wired (e.g. DSL or cable connection) connection, but maintains that because of the “unique technical and operational characteristics of wireless networks,” wireless Internet delivery should be exempt.

Most of the commentary focused on how this will impact cell phone access to the Internet - which is critically important. However, also important is the impact on wide area wireless networks. Similar to the wireless network you might have in your home or office, new wireless protocols are being developed that can cover more than your living room - providing the possibility for your home Internet connection being provided by anyone in your city - without the need for installing a wire to your home.

Between cell phone access and wide area wireless access, DSL and Cable will soon lose their status as the fastest growing way to get broad band Internet. If the most promising future method of delivering Internet access is too technically and operationally “unique” to be covered by the principle of net neutrality, then the future development of the Internet is in real trouble.

Fortunately, we have a few things going for us.

The first and most important is that we have public opinion on our side. And we should use it. This issue is not a technical one - the question is: when we use the Internet, should we have equal, non-discriminatory access to everything? Few will disagree.

Second, let’s build alternatives. Most attempts at controlling the Internet have failed miserably (consider the scrap heap of technologies to keep you from copying your music and videos). When, in an early violation of net neutrality, our service providers prevented us from sending email, we routed it through a different port. We can and will test the limits of every attempt control and profit from our usage of the Internet.

Most importantly, however, we need to join this fight. The future of our Internet depends on it.

Remarks given at WSF/Dakar Convergence Center Fundraiser

Fri, 08 Jan 2010 09:32:44 +0000

On January 23, 2011, the Indy Media Convergence Center will open it’s doors in Dakar, Senegal. Beginning two weeks before the start of the World Social Forum, the Convergence Center will be a space for collaboration and skill shares, bringing together media activists from all over Africa and the rest of the world. The gathering is a continuation of the work begun at two previous African convergence centers (Dakar in 2004 and Nairobi in 2007), with the goal of fostering deeper collaboration among African media activists and with the rest of the world.

May First/People Link plans to raise money to send one participant. The US Social Forum has allocated funding for a delegation, which will include three members of the Information, Communications, and Technology working group.

Below are remarks I made on a panel organized in New York City as a fundraiser for the event. The other panelists were Sean Jacobs from the New School, Omoyele Sowore of Sahara Reporters, Valentine Eben (aka “Sphinx”) of IMC-Ambazonia and The Way Forward Network and Mohamed Keita of the Committee to Protect Journalists.

In April, 2010, tens of thousands of people from over a hundred countries came together in Cochabama, Bolivia to draft and affirm the Cochabamba People’s accord on Climate Change and the Rights of Mother Earth.

This gathering marked a critical point in our history for a number of reasons.

The accord called for systemic change, identifying capitalism as a fundamental cause of the destruction of the planet
The leadership and full participation of the indegenous peoples of the americas was strongly reflected in both the gathering and in the final accords.
The gathering reflected the power of democratization and the free flow of information.

This last point is particularly important for all of us here.

Thirty years ago, there were fewer than a handful of countries in Latin America that could make any kind of credible claim to democracy. Now, almost all countries can. It’s no coincidence that this democratic transition brought with it a significant political shift to the left.

It’s also no coincidence that the launch of this political shift began with the Zapatistas and their innovative use of a newly massified medium called the Internet.

Despite the enormous differences between Latin America and Africa, in both regions democratizing media has an impact on political democracy and the global movement for justice. The Convergence Center in Dakar will be one contribution to this effort.

Our impact, however, goes even further with this project.

Over the last few centuries, we’ve experienced a theft, on a global level, of astounding proportions. A theft that, despite its complexities, nonetheless draws a clear line of transfer that travels from Africa to Europe, North America and a handful of other countries.

The IMF and World Bank’s response to this theft is to provide charity and loans with steep conditions to African countries.

May First/People Link’s response is different: We recognize that the technology skills and training that we’ve developed in the United States do not belong to us alone. Our skills belong to the world. Sharing what we have is a political mandate. It is required of us.

The Convergence Center in Dakar is an opportunity for such sharing.

Despite the global in-equalities, from the outset, the Convergence Center is based on real and equal collaboration. Not charity, not outside control, but a recognition that we all have something to contribute and something to learn. We’re honored to be included.

rm -rf at ~ 500 Mbytes/second

Fri, 08 Jan 2010 09:32:44 +0000

If you ever have the misfortune of accidentally passing the path to a directory containing 177GB of data to the rm -rf command, I’ll start by suggesting that you hit ctl-c. The sooner the better.

Next, assuming you have some sort of backup, you’ll be staring at two monumentally large data sets, wondering exactly what was deleted from the original.

With help from dkg, I learned somethings about rm -rf. For one, it deletes one top level directory at a time. So - a comparison of the top level directory listings of the original and backup is a good place to start.

Top level directories that are entirely missing in the original are easy to restore. However, the presence of a top level directory in the original doesn’t mean it was un-touched.

Next, you’ll want to figure out which top level directory rm was operating on when you hit ctl-c. dkg discovered that ls -UR will provide a listing in the same order that rm -rf uses. The -U means do not sort. Note - the unsorted listing of the backup directory might not be the same as the unsorted listing of the original, so ls -UR is only really helpful on the original directory.

After selecting the first top level directory, rm -rf seems to delete all files in that directory first (presumably in the same order that ls -UR will list them), then it enters the first sub directory (as returned by ls -UR) and repeats the process.

With a careful comparison of ls -UR on the original with the directory listings on the backup, you should be able to pinpoint the exact sub directories affected, allowing you to restore only the files and directories that you deleted.

Thanks to dkg for technical and blog title suggestions.

Two Terabyte disks: prepare for major changes

Fri, 08 Jan 2010 09:32:44 +0000

When we bought our first 2 TiB disk we had no idea what was in store.

Over the last several months we have been tearing our hair out over painfully slow performance caused by I/O bottlenecks compounded by our desperate attempts to remedy the situation by moving terabytes of data from slow performing disks to better performing disks.

Over time, we’ve started to piece together the underlying issues and figure out a strategy for properly using these disks - causing yet more slow downs during new server installations.

Below is an attempt to explain at a high level why previously routine tasks, such as installing Debian on new machines, or replacing dead hard disks has become significantly more complicated and taken exponentially longer than before.

Partition tables

Up to now, most of us have created partition tables (the information stored on a disk that instructs an OS or bios what partitions exists, where to find them, etc.) in the Master Boot Record or MBR. That’s the first 512-byte sector of the disk. When you run the debian installer or just about any disk partition utility, it stores the partition info in the MBR.

The MBR approach, however, has a limitation: it can’t handle disk partitions larger than 2 TiB (well… technically that means you could still use the MBR with a 2 TiB disk because no single partition would be larger than 2 TiB… however, given the rate of growth in disk size, it seems like now is the time to tackle this problem).

Fortunately, a new partition table layout has been created: GPT or GUID Partition Tables.

The bad news is that not all of our favorite tools can handle GPT partitioned disks, and some will fail spectacularly.

Grub

Although legacy Grub (Grub 1) supposedly supports GPT tables, when making this change, we’ve opted to switch to Grub 2, which their web site proudly proclaims has been re-written from scratch. Ug. Although legacy Grub users will recognize a few bits and pieces from Grub 1, it’s a steep learning curve.

When combined with GPT, that learning curve includes a significant departure from how we previously installed Grub on a disk. If you want to use Grub 2 with GPT, you need to create a small partition for Grub on the disk (in addition to your regular /boot partition) and add a flag on that partition called bios_grub. Note: Grub 2 only needs this partition if you are using a GPT-partitioned disk.

Then, when you run install-grub, it will be installed into that partition in a way that will properly boot your operating system.

Partition alignment

If all of this wasn’t enough…

With the introduction of 2 TiB disks, disk manufacturers are beginning to change the way they are writing data.

Previously, disks wrote data in 512-byte sectors. As a result, all disk utilities of the recent past have religiously created partitions and all other forms of dividing up a disk on 512-byte boundaries.

Some manufacturers of 2 TiB disks, however, are writing data in 4096-byte boundaries. That means if you create a partition that overlaps a 4096-byte boundary you are essentially screwed.

Consider a disk in which the following pipes represent 512-byte boundaries and [Pn ] represent partitions properly aligned along those boundaries:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 
[P1     ][P2        ][P3                        ]

All partitions neatly start at the beginning of a 512-byte block. Every time the disk wants to write, it can easily fit the data into the sectors.

A 2 TiB disk that uses 4096-byte sectors, however, needs to be divided along 4096-byte boundaries, displayed below with the middle row of pipes. As you can see, your beautifully aligned partitions are now a mis-aligned mess:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | 
|                       |                       |
[P1     ][P2        ][P3                        ]

What you can do

We haven’t yet fully tested mdadm, cryptsetup and lvm to ensure that they create data on 4096-byte boundaries. Initial poking around suggests that they do - but more work is needed to be certain.

The version parted we’re using (squeeze), on the other hand, will not attempt to align your partitions on 4096-byte boundaries for you. You need to do that yourself by specifying the exact, properly aligned boundaries.

We have a write up with the new steps for creating a Debian server using 2 TiB disks.

The summary is: when partitioning your disk using parted:

switch the unit to sectors (unit s)
ensure that the starting sector is divisible by both 8 and 512
ensure that the ending sector + 1 is divisible by both 8 and 512 (so that the next sector start point is properly aligned)
ensure that the size is divisible by both 8 and 512.

For the math challenged, here’s a functional layout of a GPT partitioned disk:

~ # parted /dev/sda unit s p
Model: ATA WDC WD20EADS-00R (scsi)
Disk /dev/sda: 3907029168s
Sector size (logical/physical): 512B/512B
Partition Table: gpt

Number  Start     End          Size         File system  Name      Flags    
 1      2048s     4095s        2048s                     biosboot  bios_grub
 2      4096s     1052671s     1048576s                  boot      raid     
 3      1052672s  3905974271s  3904921600s               pv        raid

You can get there with these commands:

parted /dev/sda mklabel gpt

parted /dev/sda unit s mkpart biosboot 2048 4095 
parted /dev/sda set 1 bios_grub on 

parted /dev/sda unit s mkpart boot 4096 1052671 
parted /dev/sda set 2 raid on 

parted /dev/sda unit s mkpart pv 1052672 3905974271
parted /dev/sda set 3 raid on

I’m not sure what the disk size limitations of GPT are… but I hope we don’t reach them any time soon.

video4linux and audio

Fri, 08 Jan 2010 09:32:44 +0000

With cop16 coming up in just a few months, we’ve been working at MFPL on organizing people from around the world to share live video of the COP16 related events: protests, panels, performances, etc. The goal is to use live video as an alternative to expensive and environmentally destructive travel - a goal well-suited to the environmental focus of COP16.

I’ve spent the better part of the last three weekends figuring out how to do this on Debian Squeeze using all free software and codecs.

The debconf organizers have done an amazing job developing and documenting how to broadcast live video from a conference using DV via firewire input. It’s impressive and, based on my experiences, works quite well.

The problem is the firewire port. Computers aren’t made with firewire cards any more and even if they were, I don’t have a video camera with firewire out. I do, however, have a laptop with a USB camera and a mini audio plug that will take a cheap microphone. And, there are millions of others in this position.

The other draw back to the debconf approach is that it assumes all video cameras are attached to the same local network. We are interested in having people contribute video content from all over the world.

The main pieces

Video

video4linux is a programming interface that makes this all possible. Most video-based applications available on linux support it as do most web cameras that I’ve experimented with.

Audio

Audio proved to be the most difficult piece - way harder than video. Many of my problems seemed to have been largely specific to my computer, but not all of them.

Part of the problem with audio is that, on linux, it’s a real mess. There are a half dozen methods in use for accessing your audio card. Furthermore, in my case, my audio card is not well supported in linux. More on that below. The important piece is that, whenever possible, I tried to use alsa.

Codecs

I decided to only focus on free and open source codecs. Google seems to have made VP8 free, which means we may be moving away from theora encoded ogg files toward VP8 encoded .webm files, however, for now all the Debian tools in squeeze work with theora, vorbis, and the ogg container.

Media Server

icecast is a streaming media server. The goal is to get my laptop to send an audio/video stream to our icecast server, which will then be responsible for re-distributing it over the Internet.

Media player

I’m only testing with the HTML5 video tag. With Firefox 3.5 and up, video can be displayed directly in a web browser without any extra software.

The pain

ffmpeg2theora

I started off thinking I could do everything with a simple ffmpeg2theora command piped to oggfwd. Something like:

ffmpeg2theora /dev/video0 -f video4linux2 -o /dev/stdout | oggfwd icecast.server 8000 secret /test.ogg

That is an elegant command and it worked perfectly the first time I ran it. With one problem: there’s no audio. ffmpeg2theora will gladly add audio to it’s output, provided your input has audio. However, /dev/video0 provides just video and there is no way to specify a video input and an audio input using ffmpeg2theora. Sigh.

I was so loathe to give up on such an elegant command, that I started working on sending two streams to our icecast server: one using ffmpeg2theora for video and one using darkice for audio. I wouldn’t recommend this approach - there’s no way to keep the audio and video in sync. However, I couldn’t even get that much to work due to some kind of strange bug. Seems that when I run darkice, I get a nice consistent audio stream to my icecast server. If, during this stream, I start a video4linux device (seems to affect any video4linux device, even cheese), it causes darkice to crap out and my audio input stops working. I opened the bug against cheese - who knows where it really belongs.

ffmpeg

Next, I moved on to ffmpeg - which does allow for both a video and audio input.

I could successfully get ffmpeg to record audio with the command, using the alsa-oss compatible driver:

 ffmpeg -f oss -ar 48000 -i /dev/audio -acodec pcm_s16le out.wav

It even works with alsa directly (note the need for -ac 2 - alsa fails with the default 1 channel):

 ffmpeg -f alsa -ac 2 -ar 48000 -i hw:0,0 -acodec pcm_s16le out.wav

However, when I add my video4linux2 device, I lose the sound:

ffmpeg -f alsa -ac 2 -ar 48000 -i hw:0,0 -acodec pcm_s16le -f video4linux2 -s 320x240 -i /dev/video0 out.mpg

The video plays back fine, but the audio is silent. At this point, I moved on to vlc… however, I later discovered the problem (which I will describe here, out of chronological order).

Turns out, my system is not playing pulse audio properly, or at least mplayer is not. The audio really is there. I just needed to test using mplayer with -ao alsa:

mplayer -ao also out.mpg

After much haggling with options, I finally got this train wreck to run without an error:

 ffmpeg -f alsa -ac 2 -ar 48000 -i hw:0,0 -f video4linux2 -s 320x240 -i /dev/video0 -f yuv4mpegpipe -pix_fmt yuv444p - | \
  ffmpeg2theora -o - - | oggfwd icecast.server 8000 secret /test.ogg

However, the video ran at a crawl, I never did hear any audio, and the process died after a 3 minutes. ffmpeg was not going to be an elegant solution.

vlc

vlc seemed like a perfect option, given that it runs on linux, Mac and Windows. If I could get it to work on linux, providing directions for other operating systems would be a breeze.. Beginning with the graphical user interface, I selected Media -> Convert / Save…. Then I clicked the Capture Device tab, to indcate I wanted to convert/save something I was capturing. I hit the Convert/Save button (leaving all settings at their defaults). The next stream suggested “Video - H.264 + ACC (TS)” as the profile. I left it alone. Then, I entered the path to the file I wanted to save to (ending in .mpg), and lastly clicked Start.

And… I got this error:

Streaming / Transcoding failed:
It seems your FFMPEG (libavcodec) installation lacks the following encoder:
H264 - MPEG-4 AVC (part 10).
If you don't know how to fix this, ask for support from your distribution.

This is not an error inside VLC media player.
Do not contact the VideoLAN project about this issue.

So, I repeated, but this time, for profile, selected “Video - Theora + Vorbis (OGG)”. This time it recorded. But, when I played back using vlc, it played back at twice the recorded speed, and there was no audio (no audio playing back in vlc or in mplayer using -ao alsa).

At this point, I saw that vlc 1.10 was available in unstable. In case I was experiencing fixed vlc bugs…

sudo aptitude install vlc/unstable vlc-data/unstable vlc-nox/unstable

Sadly, no difference :(. The theora/vorbis file still played back without audio and about twice the speed at which it was recorded.

Not one who gives up easily, I researched online tips for using command line vlc (cvlc) and came up with:

cvlc v4l2:// :v4l2-vdev="/dev/video0" :v4l2-adev="/dev/audio" --sout \
 "#transcode{vcodec=theo,vb=800,scale=1,acodec=vorb,ab=128,channels=2,samplerate=44100}:std{access=shout,mux=ogg,dst=source:secret@icecast.server:8000/test.ogg}"

Works for video (barely - it’s pretty choppy) but still no audio :(.

So much for vlc.

giss.tv and a ray of hope

At this point… frustration with vlc set in and, after some browsing, I came across giss.tv and their docs page.

The Webcamstream-v4l2.pys python script was the first to catch my eye.

After downloading, I tried to run it with:

python Webcamstream-v4l2.pys

But, got the error:

Error: Could not initialize supporting library. gstautovideosink.c(367): gst_auto_video_sink_detect (): 
 /GstPipeline:pipeline0/GstAutoVideoSink:autovideosink0:
Failed to set target pad

One of the helpful giss.tv folks suggested I try to run xvinfo, which returned:

0 jamie@chicken:~$ xvinfo 
X-Video Extension version 2.2
screen #0
no adaptors present
1 jamie@chicken:~$

After considerable searching and debugging (I have a Toshiba Satellite with a Radeon HD 3200 Graphics card), I finally discovered debian bug 579918 which helped me realize I needed the ^%$ propriety firmware-linux package installed. After re-booting, xvinfo reported a lot of information (and I discovered that my machine would wake from suspend successfully again).

Next problem when trying to run the python Webcamstream script:

0 jamie@chicken:~$ python Webcamstream-v4l2.pys
The program 'Webcamstream-v4l2.pys' received an X Window System error.
This probably reflects a bug in the program.
The error was 'BadIDChoice (invalid resource ID chosen for this connection)'.
  (Details: serial 836 error_code 14 request_code 1 minor_code 0)
  (Note to programmers: normally, X errors are reported asynchronously;
   that is, you will receive the error a while after causing it.
   To debug your program, run it with the --sync command line
   option to change this behavior. You can then get a meaningful
   backtrace from your debugger if you break on the gdk_x_error() function.)
1 jamie@chicken:~$

Time to give up (for now)…

More browsing of giss.tv led me to yet another program… Theora Streaming Studio - TSS.

A Debian squeeze deb was not available, so I downloaded the lenny version. Ug. It required libraw1394-8, which is not available in squeeze (libraw1394-11 is).

So… downloaded the source. It relies on automake version 1.10. 1.11 is in squeeze, so I had to run:

sudo ln -s /usr/share/automake-1.11 /usr/share/automake-1.10

Then installed tss with the standard:

./configure
make
sudo make install

It seems to have a lot of potential - but it did not seem to work for me and didn’t provide any output to let me know what was wrong.

So… I returned to the Webcamstream-v4l2.pys script. The script relies on gstreamer for handling all of the heavy duty video and audio work. Even though I couldn’t get the script to work (for X11 reasons), gstreamer seemed very impressive.

Debian squeeze ships gst-launch-0.10, a developers command line tool for testing various things that the gstreamer library can do. I created an alias in my .bashrc file so I could simply type gst-launch to invoke the problem. After reading through some man pages and a few helpful examples on the web and from the giss.tv script …

Working audio recording:

gst-launch alsasrc ! audioconvert ! vorbisenc ! oggmux ! filesink location=input.ogg

Working audio streaming:

gst-launch alsasrc ! audioconvert ! vorbisenc ! oggmux \
 ! shout2send ip=icecast.server port=8000 password=secure mount=/test.ogg

Working video recording:

gst-launch v4l2src ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=320,height=240 \
 ! theoraenc quality=16 ! oggmux !  filesink location=input.ogg

Working video streaming:

gst-launch v4l2src ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=320,height=240 \
 ! theoraenc quality=16 ! oggmux !  shout2send ip=icecast.server port=8000 password=secret mount=/test.ogg

Working combo recording:

gst-launch v4l2src ! queue ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=320,height=240 \
 ! theoraenc quality=16 ! queue ! oggmux name=mux alsasrc  ! queue !  audioconvert ! vorbisenc \
 ! queue ! mux. mux. ! queue ! filesink location=input.ogg

And, finally (!!!)…. working combo streaming:

gst-launch v4l2src ! queue ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=320,height=240 \
 ! theoraenc quality=16 ! queue ! oggmux name=mux alsasrc  ! queue !  audioconvert ! vorbisenc ! queue  \
 ! mux. mux. ! queue ! shout2send ip=icecast.server port=8000 password=secret mount=/test.ogg

Success!! My first live video and audio stream with acceptable quality.

\0/ \0/ \0/ \0/ \0/ \0/ \0/ \0/

I did some more tweaking and came up with the following, which, in addition to streaming to an icecast server, displays the video and saves it to a local file:

gst-launch v4l2src ! queue ! ffmpegcolorspace ! videoscale ! video/x-raw-yuv,width=320,height=240 \
 ! tee name=tscreen ! queue ! autovideosink tscreen. ! queue ! videorate ! video/x-raw-yuv,framerate=25/2 \
 !  queue ! theoraenc quality=16 ! queue ! oggmux name=mux alsasrc ! queue ! audioconvert ! vorbisenc quality=0.2 \
 ! queue ! queue ! mux. mux. ! queue ! tee name=tfile ! queue ! filesink location=stream.ogg tfile. ! queue \
 ! shout2send ip=icecast.server port=8000 mount=test.ogg password=secret

Flumotion

All of this gstreamer business eventually led me to flumotion, an elegant collection of programs that use gstreamer and python’s twisted library to create a full featured streaming studio. The program is GUI driven to make it easy for newbies, while at the same time, it is dreamily composed of many separate and discreet parts providing a level of flexibility that is really useful.

Getting flumotion to live stream video and audio on Debian squeeze did take some work and help from the flumotion developers via IRC.

For starters, I had to add the flumotion user to the video and audio group (and then restart flumotion). In addition, I needed the python-gi package.

I could then run flumotion-admin and work through all the default options in the wizard except overlay… which produced the following error:

gst-stream-error-quark: 1
gstbasesrc.c(2543): gst_base_src_loop (): /GstPipeline:pipeline-overlay-video/GstAppSrc:source:
streaming task paused, reason not-negotiated (-4)

I never did figure it out - I simply unchecked overlay in the wizard.

The default options, however, used a test video and test audio source - not my webcam and audio card.

When I tried to stream using my hardware capture devices, flumotion insisted that another program was using my sound card. I was sure pulseaudio was turned off and nothing else should have been accessing it. Finally, on the suggestion of one of the developers, I applied a patch to the audio.py file that is scheduled for the next release and it all worked!

Next steps

gstreamer definitely seems to be the best tool for the task. While flumotion is the best general purpose tool, the Webcamstream-4vl2.pys gave me a lot of ideas on how to create a simple program that just streams live video and audio to an icecast server. Given the work done with oggconvert to get gstreamer and python bindings functional on Windows, it even seems possible to make something that would run on Windows.

However, the biggest next step to really meet our goals will be to get a live streaming option for the android phone. Seems like at least someone is working on getting gstreamer to work on android. Hopefully that will progress!

Web casting and the Climate Change Conference in Cochabamba, Bolivia

Fri, 08 Jan 2010 09:32:44 +0000

One of my first experiences as an activist happened while I was a teenager - I went door to door surveying my neighbors in Lincoln, Nebraska to determine interest in recycling. My small contribution made me feel part of a movement, an experience that helped propel me through a life time of organizing and activism. Since then I have been heavily involved with both alternative media and free Internet issues - following a path toward a more global and revolutionary movement than I found in the environmental circles of the 80’s in Nebraska.

Now, twenty years later, the issues of the environment, media, technology and revolutionary change are coming together in a global movement unlike anything I could have ever imagined.

Global Conference on Climate Change: Cochabamba, Bolivia, April 2010

In March, May First/People Link (MFPL) was contacted by Nick, the Media Liaison for the Global Conference on Climate Change in Cochabamba. He invited us to participate on the technology team supporting the event. After some back and forth, we received our assignment: provide a free/open source live Internet video stream of the proceedings. Our contribution would compliment the non-free software based live stream provided by the One Climate team. Taking no chances, the conference organizers wanted to ensure that we would have a way for people outside Bolivia to experience the event - even if it meant having two teams working on the same issue. Furthermore, the lead tech organizer wanted to showcase free and open source software during the event.

We immediately put our team together: Mallory, Maritza and I were tasked with going to Bolivia to provide direct support. Greg, Nat, and Daniel provided on the ground server support from New York. And Alfredo was in charge of organizing an international event to support the conference: an Internet-based, multi-city, interactive, video/audio conference connecting delegates at the conference with people from around the world who could not be there in person. Unlike the video stream - which would be more like watching TV, this event was designed to allow two-way video and audio interactions between all participants.

Web casting

What is Web casting?

Most of us have watched video online - typically we stumble upon a web site with a box and play button. This technology allows us to play video on demand - meaning that when we, the viewer, control when it starts, etc. This type of video watching requires that the video itself is pre-recorded.

Live video web casting, on the other hand, means watching a video portraying an activity that is happening live. Nothing pre-recorded; nothing on demand. We lose the ability to control when we watch it; however, we gain the experience of watching something together, with the rest of the world.

The setup for web casting involves:

A cable providing a live video signal and another cable providing a live audio signal (much like the cables you might use to connect your VCR or DVD player to your TV). These cables can come from a video camera shooting a live event, or a video switcher in a TV studio, or even a Television.
These cables are plugged into a computer that converts the signal into a digital format.
That computer sends the signal, over the Internet, to a server accessible to the world.
One (or more) web sites include the familiar box and play button that, when clicked, connects the user’s browser to the server, displaying the video stream.

Our first steps

In the airport in Miami, we first heard the news of the volcano eruption in Iceland that was disrupting all European travel. Later it was confirmed: the One Climate folks, based in Europe, were not going to be able to make it - web casting was up to us entirely.

Upon arrival in Bolivia, we went to the conference site at Valley University in Tiquipaya, 30 minutes from Cochabamba. We were introduced to the local tech team - organized by Daniel Viñar, whose international nature mirrored the experience of the conference: Uruguayan by birth, his parents were exiled in France, where he grew up until he moved to Bolivia in the nineties. His team came from Paz, including paceños Freddy, Victor, Oliver, Armin, Mario, Jonathan, French ex-pats Clement, and Sylvain, and Ivan (from La Paz, studying in Peru).

In addition to us, the tech team was joined by a disparate group of web editors from around the world, some organized by Nick, others from the New York Bolivian mission (which brought 30 delegates), and many more folks who did a tremendous job editing, teaching and promoting the dynamic web site.

Our first decision for web casting was on the technology to use. We had a problem. Using free software that was controlled by the conference was non-negotiable. In addition, we preferred to use free formats and protocols. We had tested a successful web cast that used the free ogg/theora format and the free shoutcast protocol. The problem is that only people running a version of the web browser Firefox greater than 3.5 (or who had already downloaded the software needed to view ogg/theora video) would be able to view the web cast.

After some discussion, we decided that we would need to compromise on the video format and protocol, choosing the proprietary Flash format and proprietary RTMP protocol - both owned by Adoboe. A very large percentage of computers already have the software installed that is needed to view videos in the Flash format. And the RTMP protocol provides a means to send Flash videos to and from a server running Adobe’s multi-thousand dollar Flash Streaming Server.

This compromise was made possible by the free software developers who created Red5 - a fully free and open source server software that works as a replacement for the Adobe software. In addition, software developers have produced client software, like Gnash, that are free and open source flash video players.

While using Red5 and Gnash is a functional compromise now (which allows people to participate fully using only free and open source software or, if they choose, the Adobe proprietary versions), in the long run, relying on the proprietary Flash format and RTMP protocol places the movement in a precarious position. At any time, Adobe can change the protocol, which could cause Red5 and Gnash to stop working seamlessly with the proprietary versions of the software. This compromise means we are not fully in control of our software.

Thanks to help from our NY crew who setup our servers, we had several servers, ready to go, with which to expirement. We immediately went to work on Saturday, trying to wrap our heads around Red 5, which was written in a language (Java) that nobody on site was familiar with. We finally figured out the steps to properly install Red5 on Debian, and then, late into the night, after hours of frustration and on the cusp of calling it a night, Mallory discovered the magic combination of steps that resulted in a successful stream from a web cam in Cochabamba to our server in New York, and then back again to a web browser. Victory! Our job was one. Or so we thought.

We knew that we were not going to web cast using the web cams attached to our laptops. The quality would be terrible (no ability to zoom or change angles) and, of course, our laptops would need to be in the room where the event was happening. Instead, our plan was to take a video feed from the Television Team

the group responsible for broadcasting the proceedings via satellite to the rest of the world. The TV Team already had organized multiple camera crews and studios for this purpose.

The next day, Sunday, we began experimenting with the device that takes a video signal (like the kind we expected to receive from the TV team) and converts it into a digital format that we could then send to our Red5 server. The tech team had purchased several Creative SB0630’s for this purpose. This device proved to have a fatal problem: it shipped with proprietary Windows drivers only: we could not stream video with a computer running the free/open source Linux operating system.

Mallory, Armin and I followed Ivan into a cab and off we went to the Concha - the giant Cochabamba market, which was a 45 minute drive from the conference center. After walking for close to 30 minutes, through stalls of every item imaginable, we finally hit the electronics section, and then the computer section. We split into two groups in search of an alternative digital video converter. After 30 minutes, Mallory and I turned up empty handed, but Ivan and Armin discovered two possibilities. We settled on a a an Encore.

Back at Tiquipaya, I spent several hours trying to get our new device working on Linux, but without luck. Like the Creative SB0630, I couldn’t get Linux to recognize the device as a video input. The 4 hours it took to get to and from the market really cut into our work day on Sunday, and time was getting short.

Furthermore, Daniel wanted us to test our web casting capacity that night during a pre-conference interview. In a move that would be repeated throughout the conference, we turned to Gavinda from Earth Cycles, a media maker who demonstrated an uncanny ability to have every piece of technical equipment we needed in the last minute. For the test, he turned up a Pinnacle video converter and a Windows laptop with the drivers pre-installed. Although it wasn’t free software end-to-end, we nonetheless pulled together a working web stream solution just in the nick of time. We went home happy - knowing that we could at least produce a working web stream - and still with one day before the official conference kicked off on Tuesday.

On Monday morning, by the time the MFPL team had arrived, Ivan had miraculously gotten our new Encore card to work with Linux - with the help of the free software program TV Time. This was our final break through! We quickly ran a test and, with still half a day to go, we had a working, end-to-end free software based video streaming system in place.

For the second time, we thought our job was done.

Big Blue Button

Meanwhile, Alfredo was forging ahead organizing our multi-city event.

We were participating in a historic, global environmental summit - we needed a way for people to not only watch what was happening, but participate interactively with the delegates.

Our plan was to use the free/open source software Big Blue Button, which allows users to login to a web site, optionally connect your web camera, microphone and speakers, and then have a multi-person video/audio/text conference.

Alfredo’s job was to bring in the audience. Thanks to his work and the support of the Bolivian Mission in NY, we received a tremendous amount of interest. By the time Mallory, Maritza and I arrived in Bolivia, Alfredo had confirmed four cities - and not only their participation, but also their commitment to organize an audience: The Brecht Forum in NY, Encuentro5 in Boston (with on the ground tech support from MFPL member and USSF volunteer Ross), the Venezuelan mission in Chicago, and a collaboration between the Latin American Solidarity Organization and Movement for Justice and Peace in Olympia, Washington.

Keep in mind: we had run limited tests from New York, but had never tried anything like this before, either conceptually or technically.

To be safe, we organized a test-run on Sunday night. All four cities “showed up” online and the results of the audio and video part were … a disaster. Fortunately, Big Blue Button provides an interactive text chat - a format that is consistently reliable!

Via chat, we worked through all the various struggles each city had getting their video and audio working. Finally, after close to an hour, we got everyone working (and documented a BBB tip sheet for a functional multi-city conference). We all agreed to login at 6:00 pm for the real event, one hour before it was scheduled to start.

With two days before the scheduled live event (Tuesday night), the pressure was on to recruit delegates from the conference to participate on the Bolivia side. Four confirmed cities! And, since the conference hadn’t started yet, we had no idea who would be available!

Maritza went to work. With the help of the Bolivian mission (via Taleigh, our delegation leader) we started making connections.

It was slow work. On Monday we had several names and sent out many emails. By Monday evening, we had one confirmation, a Peruvian Economist named Emil Sifuentes.

By Tuesday afternoon, we had two more “maybes” and we started to worry about whether Emil would be able to make it, since we had no reliable way to confirm his participation.

Tuesday at 5:00 pm, 1 hour before our pre-event test, we learned that the studio we reserved wouldn’t be available after all and the alternate room we had in mind didn’t have an Internet connection. Through a lot of scrambling and last minute assistance from Clement, we finally got our alternative room cabled and with the help of Victor, Oliver and Freddy we got a computer moved into place. Mallory went to work configuring the audio and video while Maritza and I scrambled to get replacement speakers for the non-working ones we had planned to use (a university staff person eventually “borrowed” them from a administrator’s computer). At last, at 6:15 pm, we had all our technology working - just in time for Emil to arrive - a full 45 minutes early!

With an altitude induced migraine threatening, Maritza began prepping Emil, and then continued when Dulfredo Moyo from Accion Vida, one of our two maybes, arrived. By 7:00, our four slots were filled when Daniel and Nick joined the group. Maritza continued prepping our guests while I worked out the remaining tech kinks with our cities (every city, it turned out, was using a different computer for the live event than they did for the test on Sunday!).

Finally, at 7:30 pm we began. Maritza jumped into the role of multi-city facilitator and translator - providing a seamless transition between the conference participants and those from the other cities. Despite some technical difficulties, and brief outages of audio and video, we successfully transmitted introductions from each city, presentations from the conference participants on the Bolivia side, and lastly we took four questions - one from each city.

And… just before we ended we were joined by another city - Barcelona, Spain!

By the end of the event, we were exhausted, yet tremendously optimistic about the potential for building more effective virtual conferences.

Web casting continues….

Despite repeated requests, by Monday afternoon we still were missing the final component to make our web casting system functional: the video feed from the TV Team.

While we waited, we worked on scaling. We setup a Red5 server in Bolivia. In addition, Sylvain contacted allies in France who setup a second Red5 server. We did the same at MFPL (using our second collocation facility), while Mallory figured out how to send a single video stream to all four locations at the same time. Through some brilliant DNS wizardry (using Bind views), Daniel figured out how to direct viewers from inside Bolivia to the Bolivian server, and send viewers outside Bolivia to one of the three external servers.

Finally, Monday night, Ivan had had enough. Under his direction, we simply moved our computer station to the room next to the main broadcasting studio and pulled a line in. By this time there was no live programming (the feed was a loop of a promo) - but at last we had a working end-to-end setup - and just in time!

Tuesday morning - panic ensued. The opening ceremony (the second most important piece to webcast - with the closing being the most important) wasn’t coming in from the broadcast studio. Fortunately, at the last minute, the video feed we had been waiting for, finally arrived in the tech room, just as the opening ceremony was beginning. We quickly trucked our equipment back to the Tech room, plugged in all the cables, and at the last minute, we had the opening going out over the Internet.

Despite this success, the following two days were not spent rejoicing. Instead, we had to constantly monitor the signal - as our video source (traveling over many meters of cable) was far less reliable than we had wanted. Ivan, who had purchased a second digital video converter, finally installed a new digitizing station downstairs, plugging into a cable connection of Channel 7 - the state run channel which was also covering the event. This cable connection was provided by a satellite dish installed on the roof of the building by Entel (the recently nationalized Telecom company) specifically for the event. This backup allowed us to switch between the two sources, giving us far more flexibility in handling problems.

This approach proved critical - since on Thursday the main event of the day, the closing ceremony, was held in Cochabamba - which the Conference TV team was not covering. Therefore, we switched permanently to the video feed from Channel 7, plugging in directly to one of the Entel provide TV’s in the press room.

This setup proved fairly reliable for most of the day (with the exception of the time someone with a remote control accidentally turned of the TV). However, our last disaster struck just as Hugo Chavez was finishing his speech: suddenly, the TV image froze with the message: “Sin señal” (No signal).

Through a monumental mis-communication, Entel had started breaking down the equipment for the conference, starting with the satellite dish on the roof. With barely 15 minutes until Evo Morales was scheduled to deliver the final speech of the closing, we were off line.

Looking around the room - Ivan, Mallory, Sylvain - I’ve never seen more dejected faces in my life. Eventually, word reached Daniel - the man we all knew as the soft spoken and gentle leader of the tech team. He demonstrated his ability to adjust to the situation: I couldn’t quite follow his Spanish, but I was certain that I didn’t want to be the Entel employee at the other end of that conversation.

Although his words couldn’t bring back the satellite dish, his emotion sparked the team into action.

We didn’t have a satellite signal but … we did have a coaxial cable with an RF connector - the kind that can plug into a Television and act as an antenae. With all hands on deck, we plugged the cable into the television, strung the other end out of the window (ensuring that the inside wire made contact with the metal bars on the window), and, while grainy, we managed to get an over the air broadcast signal from channel 7. We were back online!

The closing

Listening to Evo speak made the whole experience come together. Naming capitalism as a primary cause of environmental destruction opens up a discussion virtually unheard of in the United States. Evo’s closing, which reflected the conference as a whole, grounded the discussion in the idea that the destruction of the planet cannot be fixed by changing one or two things here or there. Instead, we need an entirely new basis for relating to each other and the world.

The connection with indigenous movements, while not prescriptive, provided a powerful testament to the historic transience of capitalism. The movement to commodify and radically reduce the world and our interactions to profit is less than 500 years old - a drop in the bucket of human history.

And, if “500 years” rings a bell - it’s because the rise of capitalism has mirrored the genocide of indigenous peoples in the Americas, who, through intense struggle and resistance, continue to maintain a world view that takes a harmonious approach to the earth as a fundamental tenet.

Looking around the room during Evo’s speech, I saw the exhausted but contented faces of the other techies, who either took time off from their corporate jobs or who have shunned the corporate tech sector altogether, to build international alliances based on the principles of free and open source software. I was reminded that the entire world does not run on capitalism, even now. The growing movement of people from around the world who are dedicated to building a free and open Internet, unfettered by the demands of profit and capitalism, can provide a powerful contribution to the 21st century world imagined by the conference participants - one based on collaboration and a common good.

See the final declaration of the conference (Spanish)

WikiLeaks offline

Fri, 08 Jan 2010 09:32:44 +0000

I saw the keynote speech by one of the key technologists and organizers of WikiLeaks at the last Hackers on Planet Earth conference. Although the talk was mostly political, there was enough techie talk about encryption and anonymity that I assumed WikiLeaks web infrastructure was ready for any kind of attack.

Apparently not. All the encryption in the world doesn’t help you if you are hosted in the Amazon Cloud or, for that matter, with any host who doesn’t care for your politics.

To their credit, WikiLeaks moved to Amazon because a technical denial of service attack took down their previous Swedish host (I don’t imagine that they moved without a good reason). However, essentially they traded one form of denial of service for another one.

Today WikiLeaks encountered a new form of censorship that should make all of us shudder. Rather than being shutdown at the web hosting level, EveryDNS shutdown the wikleaks.org domain name.

Unlike most aspects of the Internet, the domain name system is hierarchical. There is pyramid - with a limited number of Domain name registrars (just “over 500” according to Wikipedia) that control all the domain names in the world. When you type a domain name, like wikileaks.org, into your web browser, that domain name must be translated into an IP address that is used to route your request to the correct server. The 500 or so registrars control this process.

So what can you do?

Wikleaks responed by registering wikileaks.ch (woops! shutdown as well), wikileaks.de, wikileaks.fi, and wikileaks.nl.

That’s a good start. But what if there were more? Here’s an idea. What if everyone who controlled a domain name volunteered a subdomain for WikiLeaks? For example: wikileaks.mayfirst.org. Just create an A record that points to the IP address 88.80.13.160.

If WikiLeaks has to change providers (and therefor their IP address again), our subdomain won’t work until we update it. On the other hand, seems like a good way for us all to really pitch in and share the risk that the folks at WikiLeaks are taking all by themselves. And, if the IP address changes, WikiLeaks only needs to leave behind a simple page on the old IP with a redirect to the new one.

Any takers?

Update: 2010-12-05

The UK Guardian has picked up this idea. Also - check out Paul Carvill’s blog where he documents the imwikileaks twitter tag detailing many others following on this and similar ideas and posts a link to WikiLeaks web site with directions on how to mirror their content.

It’s great to see WikiLeaks reaching out and asking for help from the community

there is clearly a huge number of us that want to offer our support.

And, I’m reasonably confident that the IP addresses 213.251.145.96 and 88.80.13.160 are under the control of WikiLeaks so we’re not spreading false information.

However, if you work with a political organization that is currently not under fire, now is a good time to consider publishing some form of a public cryptography key so if you are under attack in the future, people can verify this kind of information. Since news travels and is repeated so quickly on the Internet, it would not be hard for someone to post an “official” IP address for WikiLeaks that doesn’t belong to them.

This episode prompted us at May First/People Link to publish two OpenPGP keys. We work hard to digitally “sign” every piece of official May First/People Link information with one of these two keys, or with a key that is certified by one of these two keys. This approach provides everyone with the ability to verify that a piece of information supposedly sent by us really was sent by us.

Booting from a USB stick into Grub

Wed, 24 Jun 2009 17:41:17 +0000

Often when replacing a failed disk in a RAID1 setup (or in many other cases) grub fails to load. Usually it’s because the bios is expecting to find a boot loader and the various files on which the boot loader depends on the first disk in the machine. If that’s the disk that was replaced, then loading grub will fail.

One way of recovering is to boot the server from a USB stick that has grub installed and then manually tell grub how to boot from the disks that are available.

If you are reading this before you replace a disk - be sure to copy the /boot/grub/menu.lst file from the computer before turning it off. That will save you a lot of pain.

There are a lot of complicated tutorials on how to do this - however, if you don’t care about the contents of your USB stick, it’s really quite simple.

Put your USB disk into working linux computer (don’t mount it). On my computer it is typically recognized as /dev/sdb. If your USB stick is recognized differently (and/or your hard disk is /dev/sdb) then you must replace all the instances below of /dev/sdb with whatever your actual USB stick is being recognized. This is, well, kinda important.
Create a single partition. I typically do this with:
```
  sudo cfdisk /dev/sdb
```
Feel free to create a FAT32 partition if you want to use this stick on different machines with different operating systems (including Windows).
Mount the partition you just created:
```
  sudo mount /dev/sdb1 /mnt
```
Install GRUB:
```
  sudo grub-install --no-floppy --root-directory=/mnt /dev/sdb
```
That should create a /mnt/boot/grub directory.

It will also create /mnt/boot/grub/device.map, which is a list of disks on the computer it is running. This list will include not only the USB stick but your computer’s hard disk as well. For example:
```
  (hd0)	/dev/hda
  (hd1)	/dev/sdb
```
Change to:
```
  (hd0)	/dev/sdb
```
So that the only device showing is your USB stick. Then re-run:
```
  sudo grub-install --root-directory=/mnt /dev/sdb
```

Now comes the hard part. You need a grub configuration that will make sense for the server you are booting.

If you are using grub1…

Hopefully you have a copy of the menu.lst file from the server. In that case, simply copy it to /mnt/boot/grub/ and you are ready to go. Otherwise, you’ll need to craft one. Below is a sample.

  # uncomment these lines if you want to send grub to a serial console
  #serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
  #terminal serial
  default		0
  timeout		5
  color cyan/blue white/blue

  # simple setup
  title           Debian GNU/Linux, kernel 2.6.26-2-686
  root            (hd0,5)
  kernel          /boot/vmlinuz-2.6.26-2-686 root=/dev/hda6 ro
  initrd          /boot/initrd.img-2.6.26-2-686

  # here's a more complicated one
  title		Debian GNU/Linux, kernel 2.6.26-2-vserver-amd64
  root		(hd0,0)
  kernel		/vmlinuz-2.6.26-2-vserver-amd64 root=/dev/mapper/vg_pianeta0-root \
  ro console=ttyS0,115200n8 cryptopts=target=md1_crypt,source=/dev/md1 \
  cryptopts=target=md2_crypt,source=/dev/md2,lvm=vg_pianeta0-root 
  initrd		/initrd.img-2.6.26-2-vserver-amd64

If you are using grub2…

  grub-mkconfig -o /mnt/boot/grub/grub.cfg

Then edit. You might want something like:

  serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1
  terminal_input serial
  terminal_output serial
  insmod raid
  insmod mdraid
  insmod part_gpt
  set default=0
  set timeout=5

  menuentry "Debian GNU/Linux, with Linux 2.6.32-trunk-vserver-amd64" --class debian --class gnu-linux --class gnu --class os {
  	set root='(hd0,1)'
  	search --no-floppy --fs-uuid --set 7682a24c-b06f-456b-b3d4-bcb7294d81e2
  	echo	Loading Linux 2.6.32-trunk-vserver-amd64 ...
  	linux	/vmlinuz-2.6.32-trunk-vserver-amd64 root=/dev/mapper/vg_chicken0-root ro quiet
  	echo	Loading initial ramdisk ...
  	initrd	/initrd.img-2.6.32-trunk-vserver-amd64
  }

Put the USB stick into the target computer, configure it to boot from the USB stick via bios, and then you should see the GRUB menu come up.
It’s possible that your computer will just boot with your menu.lst file. In that case - congrats! See the last step below to figure out how to ensure it can boot without your USB stick. On the other hand, if it fails, you’ll need to experimentally figure out which disk has which partitions and which kernels. Fortunately grub supports tab completion which makes this job easier:
- When the grub menu comes up, pick from the menu list the most likely candidate and press ’e’ for edit.
- You should see the various lines from the stanzas for the list item you picked (i.e. a root, kernel, and initrd stanza). You may use the up/down arrows to select a line. If that doesn’t work, look for hints on the screen for how to get around. Going left and right on a given line may require Ctl-b and Ctl-f for back and forward. You also may need to use the delete key (not backspace) to delete characters
- Select a line, delete the characters from the end of the line, and then try tab completion with various options. For example, on the root line try typing simply:
  
  root (
And then tab. You should be presented with the available disks (numbered 0 and up). Try typing one of the disks and hitting tab and you should be presented with the available partitions. Continue this process until you find the one that seems right.
When you are done and you have successfully booted, you can ensure that boots will work without the usb key by installing grub on all available disks:
```
  grub-install /dev/sda
  grub-install /dev/sdb
```

###For more information

Boot with GRUB

Attaching files to nodes programmatically in Drupal 6

Thu, 08 Jan 2009 09:32:44 +0000

After much sweat and frustration, I’ve finally figured out how to tweak my tried and true Drupal import script to work for Drupal 6.

I often find myself creating Drupal sites for groups with either existing web sites (in other content management systems) or with file libraries that they want to use to import into their new site.

Drupal’s bootstrap function and node api makes it really easy to create a script that can be run from the command line to handle the import.

Here are the key components for making it work with Drupal 6.

The import script has to be located in the web directory. Since I don’t want people to accidentally or on purpose run it from a web session, I include the followig line:

	// prevent this from running under apache:
	if (array_key_exists('REQUEST_METHOD', $_SERVER)) {
	echo 'nope.  not executing except from the command line.';
	exit(1);
	}

Unfortunately, Drupal’s boostrap function will complain if it doesn’t detect the HTTP_HOST variable, so I add it here. It doesn’t matter what that variable is:

	// set HTTP_HOST or drupal will refuse to bootstrap
	$_SERVER['HTTP_HOST'] = 'example.org';

Next comes the boostrap function will brings in all the Drupal libraries:

	include_once 'includes/bootstrap.inc';
	drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

Although not stricly necessary, some modules require a $user variable, so it makes sense to create one. Setting the user id to 1 will guarantee that you’ll have access to do what you need to do. Depending on what you’re doing, you might want to use a user with less privileges to prevent any bug from destroying all your data.

	global $user;
	$user = user_load(1);

Here is the basic node creation:

	$node = new stdClass();
	$node->type = 'story';
	$node->status = 1;
	$node->uid = 1;
	$node->title = 'My Title';
	$node->body = 'My body;
	$node->created = time();
	$node->changed = $node->created;
	$node->promote = 1;
	$node->sticky = 0;
	$node->format = 1;
	$node->language = 'en';

Here’s an example of a CCK field:

	$node->field_date = array(
		0 => array(
		 'value' => '2009-02-09T00:00:00',
		),
	);

And at last, here’s the elusive file attachment code:

	$file = '/path/to/your/file.odt';

	// Get the file size
	$details = stat($file);
	$filesize = $details['size'];

	// Get the path to your Drupal site's files directory 
	$dest = file_directory_path();

	// Copy the file to the Drupal files directory 
	if(!file_copy($file,,$dest)) {
		echo "Failed to move file: $file.\n";
		return;
	} else {
		// file_move might change the name of the file
		$name = basename($file);
	}

	// Build the file object
	$file_obj = new stdClass();
	$file_obj->filename = $name;
	$file_obj->filepath = $file;
	$file_obj->filemime =  file_get_mimetype($name);
	$file_obj->filesize = $filesize;
	$file_obj->filesource = $name;
	// You can change this to the UID you want
	$file_obj->uid = 1;
	$file_obj->status = FILE_STATUS_TEMPORARY;
	$file_obj->timestamp = time();
	$file_obj->list = 1;
	$file_obj->new = true;

	// Save file to files table
	drupal_write_record('files', $file_obj);
	
	// change file status to permanent
	file_set_status($file_obj,1);

	// Attach the file object to your node
	$node->files[$file_obj->fid] = $file_obj;

Lastly, save the node:

	node_save($node);
	echo "Savied node: $node->nid\n";

That’s it. Below I’ve copied a real life working version that takes all the files in a given directory and creates a node for each file in which the node’s title is the title of the file, the date of the file is entered as a CCK date field, and the body of the node is a text version of the document (if it’s pdf, doc, or wpd).

	<?php
	/* 
	 * This script is used to manually import files 
	 *
	 */

	// edit the following two lines
	// set the path where the files you want to  import exist. 
	$target = '../import-files-from-mbox/files';

	// what user id should the files be imported as?
	$uid = 1;

	// prevent this from running under apache:
	if (array_key_exists('REQUEST_METHOD', $_SERVER)) {
		echo 'nope.  not executing except from the command line.';
		exit(1);
	}

	// set HTTP_HOST or drupal will refuse to bootstrap
	$_SERVER['HTTP_HOST'] = 'example.org';
	include_once 'includes/bootstrap.inc';
	drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

	// create a user that will be the author of the files/nodes
	// created

	global $user;
	$user = user_load($uid);

	// iterate over directory
	$d = dir($target);
	while (false !== ($name = $d->read())) {
		if($name != '.' && $name != '..') {
			$errors = file_validate_name_length($name);
			if(count($errors) > 0) {
				echo "Invalid name length, skipping: $name\n";
				continue;
			}

			// set some defaults for the file we will be importing
			$file = "$target/$name";
			$details = stat($file);
			$filesize = $details['size'];
			$mtime = $details['mtime'];
			$date_value = date('Y-m-d\T00:00:00',$mtime);

			// create the node object
			$node = new stdClass();
			$node->type = 'lib_item';
			$node->status = 1;
			$node->uid = 1;
			$node->title = $name;
			$node->body = extract_body($file);
			$node->created = time();
			$node->changed = $node->created;
			$node->promote = 1;
			$node->sticky = 0;
			$node->format = 1;
			$node->language = 'en';

			// custom node fields
			$node->field_date = array(
				0 => array(
				 'value' => $date_value,
				),
			);	

			// handle the file upload
			$dest = file_directory_path();
			// copy the file to the files directory 
			if(!file_copy($file,$dest)) {
				echo "Failed to move file: $file\n";
				continue;
			} else {
				// file_move might change the name of the file
				$name = basename($file);
			}

			// build file object
			$file_obj = new stdClass();
			$file_obj->filename = $name;
			$file_obj->filepath = $file;
			$file_obj->filemime =  file_get_mimetype($name);
			$file_obj->filesize = $filesize;
			$file_obj->filesource = $name;
			$file_obj->uid = 1;
			$file_obj->status = FILE_STATUS_TEMPORARY;
			$file_obj->timestamp = time();
			$file_obj->list = 1;
			$file_obj->new = true;

			// save file to database
			drupal_write_record('files', $file_obj);
			
			// change file status to permanent (default is temporary)
			file_set_status($file_obj,1);

			$node->files[$file_obj->fid] = $file_obj;
			node_save($node);
			echo "Savied node: $node->nid\n";
			exit;
		}
	}

	function extract_body($name) {
		$pos = strrpos($name,'.');
		$ext = strtolower(substr($name,$pos+1));
		$cmd = '';
		if($ext == 'doc') {
			$cmd = 'antiword';
		} elseif($ext == 'pdf') {
			$cmd = 'pdftotext';
		} elseif($ext == 'wpd') {
			$cmd = 'wpd2text';
		} else {
			return '';
		}
		exec(escapeshellcmd($cmd) . ' ' . escapeshellarg($name),$ret,$error);
		if($error != 0) return '';
		return implode("\n",$ret);
	}
	?>

Google Docs and politically responsible computing

Thu, 08 Jan 2009 09:32:44 +0000

I just saw an announcement on a politically smart list saying that a collaborative document was just made available on Google Docs (a web application provided by Google that allows multiple people to collaboratively edit a word processing document).

Many people are concerned about privacy in the realm of corporate services like Google Docs (Google, and anyone with legal access to Google, can read everything you post).

While I agree, I’m even more concerned about reliability. I don’t believe we should count on services provided by corporations (or corporations themselves) being around for the long haul and we certainly can’t count on them to gracefully close. When capitalism is doing well, corporations are purchased, merged, and re-organized with unpredictable changes in the quality or future of services. In times like the present, corporations slash services or just go belly up. And regardless of the economic climate, network access to corporate services can be turned off in an instant in response to any number of spurious copyright complaints or politically motivated criminal investigations.

What happens to our library of documents when this happens? Can we count on a warning that would allow us to retrieve our data?

You could argue that Google is one of the most financially powerful institutions on the Internet and is therefore highly unlikely to be bought or go belly up. Yet, do we really want to hitch our political movements to the financial success of a corporation whose intent is to dominate the Internet? Do we want to support a practice that will, over time, make us increasingly interdependent on Google?

Fortunately, there are alternatives. “Alternative” in this sense doesn’t mean a nonprofit or other entity offering the same services as Google Docs (yet another unreliable data silo), but instead alternative in the real sense of the word.

The project is called oo2gd (OpenOffice to Google Docs). It allows you to synchronize your documents from OpenOffice, a free office suite, to Google Docs (and several other network platforms). With oo2gd you can use Google Docs while always preserving, off line on your own computer, copies of your work.

Ikiwiki upgrading and disabled comments

Thu, 08 Jan 2009 09:32:44 +0000

Although not nearly as hairy as helping David Swanson upgrade upgrade his Drupal blog, the ikiwiki upgrade from version 2 to version 3 has been a bit hairy.

My biggest difficulty is that my published version of the blog is running Debian Etch (and pulling ikwiki from Debian Lenny) while my laptop is running Debian Squeeze. Hence, my laptop pulled in ikiwiki 3, which as prompted me to upgrade my blog, whereas my published wiki is still running 2.53.

My temporary solution to this problem is to disable commenting and make my published blog a plain rsync’ed version of the version on my laptop. Sorry! Commenting will return soon.

Installing Debian via qemu

Thu, 08 Jan 2009 09:32:44 +0000

I’ve been experimenting with kvm and qemu and run into some conceptual and practical problems. Here are some of my discoveries and short cuts.

I started by downloading the netinst Debian install image as debian-503-amd64-netinst.iso. Next I created an empty image:

	qemu-img create -f raw default.img 3GB

This file will hold the virtual server.

Next, I launched the virtual server, booting from the debian installer, passing the file I just created as the first hard drive:

	 qemu-system-x86_64 -boot d -cdrom debian-503-amd64-netinst.iso -hda default.img -curses

This command is like booting a server with a single hard drive and the Debian install CD in the CD ROM drive (and configuring your bios to boot from the CD).

I pass the -curses option to tell qemu to not try to use the graphics card (since I’m doing this over ssh).

And here’s the real trick: Since the Debian installer tries to use frame buffers, I just got “640 x 480 Graphic Mode” on my screen. Following the Debian manual, I hit “Esc” on my keyboard and was magically dropped into text mode. Next I typed:

	install fb=false

to launch the installer and ensure that frame buffer mode was disabled. From that point on I was taken through a text-based installation.

Everything went smoothly until the Grub install where I seemed to be hit with a qemu/grub 1 bug.

I killed the qemu process and made a bootable grub iso:

	mkdir -p grub/boot/grub
	cp /usr/lib/grub/x86_64-pc/stage2_eltorito grub/boot/grub/
	mkisofs -R -b grub/boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table -o grub.iso grub

Those steps created a file called grub.iso. I then booted with that file:

	 qemu-system-x86_64 -boot d -cdrom grub.iso -hda default.img -curses

And was dropped into a grub shell. I typed:

	root (hd0,0) 
	kernel /boot/vmlinuz-2.6.26-2-amd64 root=/dev/hda1 ro
	initrd /boot/initrd.img-2.6.26-2-amd64

And finally typed: boot and behold the installation booted!

Now the problem is… how do get grub properly installed??

Well… the solutions seems to be to install grub2. Before I could do that, I had to figure out how to get networking properly setup. Thanks to some help I changed the /etc/network/interfaces file on the host to:

	auto br0
	iface br0 inet static
		address 209.234.253.26
		netmask 255.255.255.224
		gateway 209.234.253.1
		bridge_ports eth0

And edited /etc/qemu-up to:

	sudo -p "Password for $0:" /sbin/ifconfig $1 0.0.0.0 promisc up
	sudo /usr/sbin/brctl addif br0 $1

Then, I restarted the virtual server with:

	 qemu-system-x86_64 -boot d -cdrom grub.iso -hda default.img -curses -net nic -net tap

I went through the same grub business. Then, from inside the virtual host, I configured /etc/network/interfaces with a real IP address.

With network access, I installed grub2 and then ran:

	grub-intall /dev/hda
	udpate-grub
	shutdown -h now

And then restarted the virtual server with: qemu-system-x86_64 -hda default.img -curses -net nic -net tap

grub2 is graphical - so I can’t access it during the bootup (there’s probably a text mode command for grub2 somehwere).

At this point I finally got around to reading the README.Debian file in the qemu docs directory and learned about the Debian command qemu-make-debian-root. Wow. That’s makes things a lot easier.

According to the docs you can run:

	qemu-make-debian-root 3000 lenny http://ftp.debian.org/ debian.img

To use deboostrap to create a debian image. Then, launch with:

	qemu debian.img -kernel /vmlinuz

Haven’t gotten this to work though…

And another thing - I realized that I didn’t have kqemu installed (check for /dev/kqemu). I installed it with:

	aptitude install kqemu-sources

And then ran:

m-a

To build the module.

Followed by:

	modprobe -v kqemu

To load the module.

Internet security and oppression

Thu, 08 Jan 2009 09:32:44 +0000

Oppression - particularly race and gender based oppression - has a long sordid history. One important aspect of oppression is exclusion. It doesn’t matter whether it’s officially acknowledged (segregation) or hidden (old boys networks) or something in between. It also doesn’t matter whether it’s explicitly based on race or gender or if that exclusion happens in a de facto way. Exclusion, in all forms, is a central tenant of oppression.

Tech culture within the Internet is one of the most gender and race-based exclusive cultures in the mainstream today. Sadly, the radical tech sub-culture of the Internet rarely challenges this exclusion and often furthers it.

Consider security. Before getting into the tech aspects of security, the general concept itself is fraught with gender and racial overtones. For example, there’s the aggressive macho approach to security, or the racialized circle the wagons to defend against the "outsiders."

Our impression (and reaction) to security can vary wildly depending on our experience with oppression. It frequently comes as a surprise to one person that an idea about security could be interpreted as oppressive. However, to an entire group of people who have experienced a pattern of exclusion resembling that approach to security, it’s profoundly racist or sexist.

Security and the Internet: default closed

With regard to Internet, there are an many ways to interpret security. However, within radical tech culture, most of the approaches involve exclusion.

Encrypted email, for example, requires that every participant have a special key, a working and configured program that can use the key, and access to this working installation for all email communications. If you don’t have that, you can’t participate.

Key-based shell access to a server is another example. A common and secure way to provide access to servers is to use an approach requiring every user to have a public key - rather than a password. You then need to connect from a computer that has the private key corresponding to your public key. More secure - yes, however it has the effect of excluding everyone without a key or the knowledge to generate one.

However, the example that is the most striking is the way we treat the concept of privacy. In radical tech culture, privacy is often the default. We decide to be private and then, maybe later, decide what we want to make public. Email list archives become private, email has to be encrypted, wikis have to be password-protected and the list goes on.

This approach requires a pre-defined group of people who are in. Nobody gets access to what this group produces unless the group decide later to make it public. And - people can only join if someone in the group can vouch for them.

Furthermore, even within this group, the overhead involved in communication is immense since every channel has to be secured before you can begin.

The result is an exclusive enclave within a network that is internationally famous for being decentralized and open. A country club on the Internet built by people who identify with left politics. And a very difficult environment for including new people.

And when a group like this is predominantly run by white men - or any other grouping of people with privilege - it effectively creates an environment that is not only exclusive but throws huge structural obstacles to challenging racism and sexism by removing itself from a bigger movement for liberation.

A “default open” approach to security

There is, of course, a need for security when organizing. However, “default closed” is not the only way to organize and, from both an anti-oppression and pro-security perspective, is not necessarily the best way.

A different approach is to consider a default open approach, and then figure out where to make exceptions. In other words:

Organize most activities in such a way that all newcomers can be safely treated as part of the group until they demonstrate otherwise.
Organize most meetings so that they are public and open to all (if it’s an online - you don’t need a GPG key or an SSH key to participate - although you will be encouraged to get one moving forward).
Publish as much information about what you are doing publicly so new people can decide whether they agree with what you are doing before deciding to participate.

This approach probably sounds familiar to veteran organizers, since it’s an old method for movement building. It also may sound familiar to techies used to Unix - since it’s the approach taken with the file system (all files are readable by everyone, except the small handful that should not be public).

In an organizing context, certain information and certain discussions will need to be exclusive to a smaller number of people who are trusted with the group - however, this division and explanation can be public without divulging the private information. And this distinction should be a healthy debate within the group.

This approach doesn’t guarantee an oppression-free organizing environment. However, it does provide an environment in which challenging oppression can take place.

In addition, this approach makes security significantly easier. Rather than trying to secure everything, we only have to focus on a securing a smaller subset of information and communications. And, we reduce our risks of divulging private information due to a technical compromise by operating mostly in the public.

Deciding on an approach to security

Ultimately, a group’s approach to security will depend on the group’s priorities. There are organizing campaigns now (and from our history) in which individuals face serious persecution for having their identities divulged or activities public. Sometimes that happens when we are working in a regime with laws we do not respect, other times our politics dictate acting in a way that makes us unacceptably vulnerable. And there are a many other reasons why staying under the radar is critical for one’s organizing objectives.

How a group of people decides to organize will be a result of the individuals involved and their collective politics. Ultimately, we have to consider the risks involved, the sacrifices we are willing to make, and what our political priorities are. While the decision will be different for every group, there are few areas of work more in need of attention to exclusion and oppression than the Internet.

Key transition

Thu, 08 Jan 2009 09:32:44 +0000

The time has finally come.

After putting it off, I’ve finally started transitioning my gpg key.

I’ve published instructions for how to find (and sign) my new key. Thanks to dkg and micah for the help in getting this out.

If you have a gpg key, you may want to check to see if your key should be transitioned as well.

Open Video Alliance 2009

Thu, 08 Jan 2009 09:32:44 +0000

Here are some of my notes from attending the second day of the Open Video Conference in June 2009.

##Opening Remarks##

Jonathan Zittrain gave the opening address. Striking was the prediction (from the conference organizers) that by 2013 90% of all video traffic will be video. We can expect bandwidth costs for providers will go through the roof. Jonathan point’s is the the big content providers ultimately sign contracts with the big Internet service providers. The basis is: the content providers suddenly, once they become big enough, can approach the ISPs and say: your subscribers want our content - let’s strike a deal. And the deal is that they don’t have to have their costs sky rocket as their bandwidth goes up.

Where does that leave small providers? It’s an impossible to sustain growth model.

Makes me think a lot more about varnish - a web proxy server that allows you to create a network of servers proxying static content (like video) across many different servers on different providers. As DSL and cable providers are rolling out very high bandwidth personal packages - it could provide an opportunity for our members to contribute their own home bandwidth to the organization for distributing our video bandwidth.

##Free Editing Software##

A new one!! pitivi. Still doesn’t work for me - but the demo makes it look like the uber simple video editing software of my dreams.

##DMCA Take downs##

YouTomb reports that it seems like the number of YouTube videos being pulled down is going down, but in fact the truth is that they are coming down so quickly after going up that they can’t track it. The average is 8 days.

In one case, a YouTube user’s video was taken down after a big company used his video. YouTube assumed the big company’s work was the original.

We were also entertained by Scott Smitelli’s experiment with YouTube’s fingerprinter. YouTube maintains a digital fingerprint database of copyright works and automatically scans uploads against this database and automatically takes them down.

Some of the conclusions about how you can fool the fingerprinter:

Don’t bother changing metadata, title, description etc.
Altering pitch or speed works (as little as 3% slowdown or 4% speedup)
Volume changes don’t work
Removing parts of the song works in most cases.

As for video:

Inverting colors works
Removing initial a few seconds works
Speedup/slow down works

Chilling Effects was mentioned as an important resource in fighting DMCA take downs.

A striking point made on a few occasions is that copyright law is causing increasing numbers of people to build private networks - a terrible trend for a public network.

HTML 5 and the video tag

Yes - it’s coming! Support is coming soon in Opera, Safari, and of course Firefox. Soon, we’ll be able to include streaming video in our web page by using the <video> tag just like we use the <img> tag for an image file. And, it looks like there’s broad support for ogg/theora as the default encoding.

The problem, of course, is that Internet Explorer doesn’t support it. How are we going to make this transition given their (still) majority share of the browser sphere?

And one last tidbid on the topic of ogg: A web site that will convert your video to ogg.

Bittorrent and pirate bay

The closing drew a comparison between the seizure of Elite Torrents in the United States, in which network admins went to jail with little or no notice among the rest of the country. In comparison, when Pirate Bay came under pressure, there were demonstrations and even a political party that jumped in to support the freedom of sharing.

Support the an important NYC conference: The Grassroots Media Conference

Thu, 08 Jan 2009 09:32:44 +0000

One of the best conferences in New York City is looking for fresh volunteers! Anyone who feels strongly not just about grassroots media but about broad based progressive organizing should consider helping out. For more information about the project (if you are not already familiar with it) - you can checkout their website.

Here’s the call for the volunteers:

CALL FOR PARTICIPATION: Sixth Annual NYC Grassroots Media Conference!

Host: NYC Grassroots Media Coalition
Date:Wednesday, January 14, 2009
Time:7:00pm - 8:00pm
Location:NACLA office
Street:38 Greene Street, 4th floor (corner of Grand Street), SoHo

Come out and help organize the most important media gathering in NYC! Get to
know grassroots media makers and social justice organizers from around the
city while working to change our city's media landscape. Network, learn
about media, and make friends!

We're looking for people to join our core organizing team. Starting in
January, you'll work closely with staff and other organizers to make this
the most diverse and exciting GMC yet. We particularly need people with
event organizing, design, and web skills, but we welcome everyone. So come &
learn about the organizing process and meet the rest of the group.
The NYC Grassroots Media Conference organizing committee actively seeks
participation from different ethnic and racial backgrounds, sexual
orientations, classes, and physical abilities.

Phone: 8023098146
Email: jbatten517@gmail.com

The politics of video distribution on the Internet

Thu, 08 Jan 2009 09:32:44 +0000

Technology, in particular the Internet, has revolutionized the use of video among progressive activists over the last 15 years. During this short span of time, the production and distribution of progressive video has moved from the domain of a small number of individuals and organizations with specialized skills and equipment to being within reach of nearly every movement activist or organization.

Despite the enormous contribution of the left toward democratizing media in general, and the influence of the left on the Internet itself, the first wave of this revolution on the Internet has corporate roots. In 2009, the ubiquity and ease with which anyone can distribute video to a large audience boils down to two develpments:

Common file format. After years of struggle and confusion, the flash video format (.flv) has emerged as the dominant standard for delivering video on the Internet. The emergence of a dominant file format has lead to the wide adoption of flash video enabled web browsers, relieving video producers of the need to save their videos in multiple and often confusing formats in order to ensure that their work is accessible to everyone. Flash video is a foramt created and controlled by the Adobe corporation.
Internet server infrastructure to distribute videos. Despite Free Speech Video’s initial foray into hosting video (which was well ahead of its time), YouTube led the way for the masses to upload and share their videos. They’ve been followed by Blip TV and other corporations who have used their capital to finance large scale, centralized and tightly controlled spaces for activists to distribute their video files.

The second wave of this revolution is yet to come. Given the economic crisis, and the growing strength of the progressive movement both in the US and internationally, now is a critical time for the left to define and develop the next wave in video distribution.

Topics

Accessibility

Video collectives have a long history and intimate knowledge around the issue of accessibility to video. Having experienced the transitions from 3/4", to Hi-8 to MiniDV and other formats, and havig gone through the expensive and time consuming process of archiving work for future generations, video collectives have a lot to contribute toward the debate over how we should be saving our work online. Who should define the video formats we use? How do we ensure they will stay free and open?

Ownership/Control

In an era where exact duplicates of video works can easily be made, how should we define ownership of our work? How does Internet distribution affect ownership? Under what conditions should we be distributing our work?

Infrastructure

Who owns the airwaves becomes who owns the servers? How do we assert control over the means of distribution? How do we communicate this principle to our allies?

Collaboration

Groups like Paper Tiger and Indy Media have redefined the way we can make television by introducing a collaborative approach. How can we extend this idea from a small group of producers to the Internet? How can Internet distribution contribute to collaboration?

Sustainability

The corporate Internet is a graveyard of failed ideas. How can we build a distribution system that won’t fail when it runs out of venture capital?

What’s happening now?

There are a number of promising developments in the area of Internet/video that are relevant in this discussion.

Internet Archive. The Internet archive is a nonprofit building a library of online artifacts (including video).

Miro. Miro TV is an effort to combine a free media player with free software that aggregates existing video on the Internet, providing a “TV Guide” for leftist video on the Internet.

Engage Media. Engage Media is an activist video sharing site focused on Asia Pacific (based on Australia). They are an example of a YouTube alternative for activists.

Transmission.cc. The Transmission Network is an international coalition of groups working on online video distribution tools for social justice and media democracy.

Open Video Alliance. The Open Video Alliance seems similar in scope to the Transmission Network. They are having a conference in July in NYC.

Americas Social Forum Inaugural Post

Tue, 08 Jan 2008 09:32:44 +0000

I’m in the LA airport awaiting my flight to Houston and from there on to Guatemala City for the third annual Americas Social Forum.

I can’t think of a more political intense and overhelming time to be talking politics in Latin America. We have the crisis in Bolivia and the global economic crisis. We have country after country electing leftists governments - with El Salvador poised to be the next early in 2009, taking the trend from South America into Central America.

And, we have a burgeoning global communications network with an immense potential for international organizing.

I’ll be posting here over the next 10 days about my experiences volunteering, participating, and organizing May First/People Link’s own workshop contribution.

Stay tuned!

Americas Social Forum: Day One

Tue, 08 Jan 2008 09:32:44 +0000

I’ve arrived, with less than an hour of sleep. The irony of the last leg departing from George Bush Airport is not lost on me.

The big news today: Evo Morales is going to address the Forum!

It’s happening on Thursday. It’s throwing a bit of wrench in the scheduling, but there’s definitely excitement here.

Beyond Centralization

Tue, 08 Jan 2008 09:32:44 +0000

We have a lot of work to do on the left to develop our political understanding of the dangers of relying on corporate Internet services for our political organizing work. However, the corporate part of the equation is only half the problem. The other half is centralization - a problem that affects even the most politically radical hosting providers.

A group of people recently began working together on the monkeysphere - a project to broaden the use of OpenPGP in our Internet work. Although progress on the project goals has been slow going, we’ve done a lot of thinking about how to collaborate in a way that avoids centralization.

(These ideas for organizing are based on a transparent organizing model - in other words, there is no space for private communication.)

Consider your typical project (a tech project or even a non-tech focused political project): First you get a web site (centralized), then you get an email list (centralized on a single list server, which sends email to people via their centralized mail provider). If you are particularly tech savvy you might also throw in a (centralized) wiki for collaboration or a project management site (like trac or basecamp). If you are working on a software project, you may want to use a centralized revision control system (like subversion). In other words, just about every aspect of the collaboration relies on a technologically centralized system.

What’s right with this model? For one, joining the project is technically easy

you look at a web site (piece of cake) or join a list (easy for most people comfortable with the Internet).

What’s wrong with this model? The people who control the central resources control the project. Often, joining a list requires moderator approval. Accessing a password-protected web site requires that someone create your account. Usually, the people who are responsible for these duties are responsible because they happened to be the ones who set the technology up - not because a political decision in the group was made to empower them with this decision-making responsibility.

In addition - what happens when the web server breaks or is seized? Or when the email list goes down? Or when any of the central resources get hit so hard with traffic (legitimate or not) that they can’t cope?

And finally - what if the politics of the group is decidedly anarchistic or otherwise politically committed to decentralization?

With the monkeysphere project we decided to experiment with collaborating using tools that were de-centralized. We started by using a tool called git. Git is a de-centralized revision control system, which is a fancy way of saying: it’s a tool that keeps track of text files - saving all revisions and changes made by all participants (like a wiki). One way git is special is that it does not rely on a centralized server for people to collaborate. Instead, each participant publishes their own copy of their files, and every other participant can choose to merge their repository with everyone else’s repository. All changes are kept track of and can be identified by author, undone, or accepted.

In the monkeysphere project, git is about as far as we’ve gone with this theory. Below, I’ve hashed out some ways to take it even further.

Web site: it is useful to have a single web site for people to go to in order to learn about your project. If you are using git, then you can publish your web site files via git. That way, every member of the project has a copy of the site in their git repository and has the capacity to publish it as a web site. Most groups would want to choose a single individual to take responsibility for this duty - however, if the group decides the chosen individual is not doing it reliably, or the chosen individual’s public web server goes down or is seized, it is technically simple for any other individual in the group to re-publish it somewhere else.

What about communication? If every member has a published blog capable of tags

we could communicate with each other by publishing blog posts with the agreed upon tag. Each project member would be responsible for pulling in everyone else’s RSS feed of their blog on a regular basis. Sending an “email” to the group would simply be a matter of posting a new item to your blog with the appropriate tag. New members join the communication by subscribing to the other members RSS feeds. A list of participants RSS feeds could be stored in the git repository.

At the moment - these ideas are barely within the reach of a fairly sophisticated group of technologists. To expect the left to adopt these strategies now is unrealistic. However, this model of organizing suggests some new core competencies that we may want to consider developing - including the use of revision control systems, RSS and blogging so we can plan for a future when de-centralized organizing can happen on the Internet.

Bolivia getting support

Tue, 08 Jan 2008 09:32:44 +0000

If you live in the United States you’d be forgiven for missing the story.

To find out what’s currently happening in Bolivia, check out the story on nacla.org, which provides the basic information and some background on the right wing attempts to over throw the democratically elected Evo Morales.

Portside also carried a couple short pieces about how the other left-leaning governments in the region are all coming to the aid of Evo Morales.

Update (2008/09/15): Looks like the mainstream press is starting to cover the events in Bolivia. There’s a Newsweek article reprinted on Portside and a reasonable NY Times article.

Update (2008/10/02): Thanks Jeff for the link to the Democracy Center coverage.

CISPES Under Attack

Tue, 08 Jan 2008 09:32:44 +0000

One of my earliest political experiences with the US government meddling with political movements was during the 80’s. I was working with Youth for A Nuclear Freeze. I was on the Latin America Committee (yes, we had some growth naming issues). We talked a lot about the FBI Internal Security investigation into CISPES (the Committee in Solidarity with the People of El Salvador).

Now, they are at it again. Please read their advisory and take the steps asked at the end.

Collaborating with git

Tue, 08 Jan 2008 09:32:44 +0000

In a previous blog I reviewed the basic steps required to setup a git environment designed to share with others. After following the steps in that initial post, you should end up with: a initial developer who has a local repository and a remote, public repository and a second developer who has cloned the initial developer’s repository and published their own repository with (if they so desire) changes to the original code submitted by the initial developer.

In this post I want to cover the following logical next scenario: how does the original developer review changes published by the second developer and incorporate them?

In addition to printing generic commands, I’m also including real-working examples from a project called mfpldues (which is a very simple project providing links to Paypal for people to use to pay their May First/People Link dues online).

The initial developer is me and has my public repository is published at:

http://current.workingdirectory.net/projects/mfpldues

The second developer is dkg and has his public repository published at:

http://lair.fifthhorseman.net/~dkg/git/mfpldues.git

The first step is that I need to know about dkg’s public repository (in this case it was published on May First/People Link’s issue tracking system).

With this information, I can add the repository using the remote add command:

	git remote add dkg http://lair.fifthhorseman.net/~dkg/git/mfpldues.git

The “dkg” part is me assigning the repository a nickname that I will recognize.

I can check what remotes I’ve added with:

	git remote

Next, I can pull in dkg’s repository with:

	git fetch dkg

Then, I can see what differences dkg has done with:

	git diff dkg/master

The git diff commands is comparing what is currently checked out (my local repository) with the dkg/master branch. “master” is the default branch, and dkg/master means the master branch in the dkg remote.

Next, I might want to switch to dkg’s branch to see if it works for me:

	git checkout dkg/master

I can always switch back with:

	git checkout master

At this point, I decide I like dkg’s changes, so I want to merge them into my master branch.

	git merge dkg/master

This will pull in the changes and commit them to my local repository (maybe adding –no-commit would be a better practice). If all is working and nice, then I publish to my public repo with:

	git push

All done!

DrupalCamp NYC 2008

Tue, 08 Jan 2008 09:32:44 +0000

Drupal scalability

I wasn’t taking notes during the CRM/Drupal workshop - but I did with Drupal tuning. Unfortunately, the focus was on hardware tuning. Boo. Much less useful I think. However, we did spend a short period of time on software tuning, which I think is the most useful.

The first question: What’s the bottleneck?

The workshop presenter seemed to think the Database was the biggest bottleneck (I’m much more familiar with php/processor being the bottleneck).

If the bottleneck is the database, the Devel module will show which queries are the slowest, which would allow you to add index or find the modules that are killing the site.

In some cases, switching tables that are high read, low write from the default MySQL table MyISAM to Innodb can help.

From the web server perspective, the Firefox Why Slow? Plugin can provide a lot of good ideas, such as aggregating css and javascript.

Other resources to try include 2bits.com and Tag 1 Consulting.

Clustering

Aha! An audience member finally pointed out the MySQL feature I’ve been dying for: MySQL proxy. MySQL proxy allows you to configure your web application to use just one database - the mysql proxy. The mysql proxy, in turn, can direct write requests to one database and read requests to other databases. In other words, that means you can use traditional MySQL cluster approach (with one central database that replicates to many distributed, read-only databases).

Memcache

Most of the session was about memcache - which, IMO, seems to be a solution that works best in combination with throwing a lot of hardware at the problem.

Reverse Proxy

Another option is reverse proxy, which caches incoming connections. Nobody reported using Squid, but one member reported success with nginx.

Multi-Media

Big news for me: in the pre-workshop chatter one of the presenters said that the image module is eventually going to be replaced by imagefield.

Example video sites:

It’s not hard to do local video, but the presenters emphasized using YouTube or BlipTV to host the video. This is pretty unfortunate. It was not a politically-based presentation, so the reasons discussed were technically-based without political thought.

If you do host locally, you must use ffmpeg for encoding the video.

What do use?

Video module is pretty much dead.

Embedded media field allows you to drag and drop videos from YouTube, BlipTV or another provider to your submit form in Drupal. View slide show gives you a lot of options for how to display the video (apparently the plain ole’ views module has some ajaxy stuff for displaying as well). With embedded media field, you will be using the provider’s player. You can use local media files and your own player as well, however, it’s a little buggy.

A better way to do local media is to use Filefield plus jQuery Media player. jQuery Media player takes a link and automatically converts it into a player.

And, Drupal Media Player is being developed that will allow you to pull in sources from other providers but play with your own player. The player is being developed in OpenLaszlo, a free language that outputs in either javascript or flash. Unfortunately, to play flash video, you need to output in flash, so Drupal Media Player will be a flash application.

Media Mover will pull in media from other locations (like an email box or an ftp site) so you don’t have to use the web upload interface. It will also push media to other locations.

Combined with a service like TubeMogul allows you to push your video to multiple sites.

All of this makes me think that the best public video strategy might be to scatter your video to various corporate sites, all the while keeping a copy yourself. Then, visitors view video served by the corporations (and they get hit with the bandwidth). Meanwhile, you always have your own copy and provided there are enough providers with your video you can easily recover if one site goes down. You can always revert to serving the copy on your own server as a last resort.

Unit Testing

Ah. This is good. The subject of this session was how to build tests that can be run automatically after you make a change to your code base to see if you broke anything.

It seems to be based on Simple Test module. There’s a manual page which appears to be based on Drupal 6/7.

The answer to: Can I use this with Drupal 5 is: yes, but it doesn’t work so well. It should work well with Drupal 6 and it’s in core for Drupal 7. I did get distracted by the Simple Test and Simple Test Documentation, which suggests simple and quick ways to setup testing for non-Drupal projects.

Email Scam

Tue, 08 Jan 2008 09:32:44 +0000

I received a rightfully concerned email from a group asking if I was the one that made the newegg purchase referenced in the email. The email (as displayed in my text mail program) looked like this:

	From: NewEgg Support
	To: info@xxxxx.org
	ReplyTo: lackey@deadbeats.com
	Sent: Jun 5, 2008 3:20 PM
	Subject: You order in process!

	Good day, info!

	Thanks for you order!
	ASUS 20X DVDąR DVD Burner with LightScribe Black SATA Model DRW-2014L1T - Retail $919.45

	You can check order status at the folowing link:
	http://www.newegg.com/mi?l=I6WC70R53692IL9KF74YV52213EL9N975LD3QS <http://www.newegg.com.id130UOZ27719H368VP0.102354124.cn/sen/index.php>

	Best regards,
	Support NewEgg.

My response via email got canned as spam. Sigh. So here it is via the web:

	I think this is an email scam.

	The link in the email that you forwarded is:

	http://www.newegg.com/mi?l=I6WC70R53692IL9KF74YV52213EL9N975LD3QS
	+<http://www.newegg.com.id130UOZ27719H368VP0.102354124.cn/sen/index.php>

	The first part looks like a legit newegg.com URL. The second part (in angle
	brackets) is typically the part that you will be re-directed to if you click
	on the link. It specifies a different server:

	http://www.newegg.com.id130UOZ27719H368VP0.102354124.cn

	The .cn top level domain belongs to china. If you go to that page, it
	redirects you to a different page:

	http://106384234523.cn/ulp/check.php

	That pages displays nothing (but, based on the name, might record the details
	of your IP address or browser).

	The web page http://106384234523.cn seems to belong to a consulting firm.

	I would suggest calling newegg.com to report the email and to ensure that
	nothing was actually billed - but the whole things seems pretty fishy to me.

	Jamie

Forget Client - Server. How about Spokes - Council?

Tue, 08 Jan 2008 09:32:44 +0000

What if we were to design a web-application (say, for example, a blogging program) that, from the start, was designed with the same goals that we would design a mass political organizing campaign? Not a small, local political organizing campaign but a massive, global campaign.

Let’s say:

Redundancy: no person or computer should occupy such a central role that if they were to be removed (COINTELPRO, network outage, etc.), the entire project would come crashing down. A redundant web-application would require us to have multiple servers on multiple Internet connections running the application in such as way so that if one fails, the application continues functioning. Ideally a server or link could go down without any loss of functionality, but I would be perfectly satisfied to tolerate a period of partial functionality provided there was an automated way to fail-over to backups within a reasonable time period. The fabled 99.999 uptime is not a political goal (although it might be a functional goal for a particular application).
Scalability: the project should be designed so that it can grow from 50 people or computers to 5,000 people or computers without needing to be significantly re-organized. Scalability can clash with redundancy. The best way to achieve redundancy would be to have every server able to run the entire application by itself (or operate with pure consensus in which all decisions are made by all people). Then, you can lose anyone (in fact you could get cut in half) and still function. That approach, however, is not scalable. What happens when you get more information than a single brain or hard drive can handle? Also - if everyone has to stay in sync, the more time and resources we spend keeping everyone and every server in sync (imagine trying to synchronize data bewteen 500 servers or do the formal consensus process with people who can’t fit in the same room). To be scalable we need a system that can delegate.
Security/Ownership balance: people and computers should be allowed to join the project and immediately have enough responsibility to feel a real sense of ownership while not having the power to bring down the entire project. A wise person once said: The only way to tell if someone is trustworthy is to trust them. Unlike in many physical situations, when working with data we have enormous opportunities to achieve this goal, provided we limit ourselves to public information. In our hypothetical blogging application, there is no private data. All data entered into the system is intended for public consumption. In this model, provided we carefully setup our system to securely preserve all versions of all data, we can invite anyone to be a server administrator. The worst thing that they can do is modify or delete a portion of the data, in which case we roll the data back to it’s original state and then cut off that administrator. Furthermore, the whole process can be publicly documented (since all the data is public already).

At the moment, there are precious few web applications that come near these goals (to be fair, there are precious few political organizing campaigns that adhere to these goals either). Most of our web applications (like blogs, or Drupal web sites, or wikis, or web-based databases) rely on a single server which, if removed, would cause the application to fail. Similarly, once a web site gets too many users, it can be very difficult to resolve that problem short of buying a new and more powerful server. Few of our currently designed programs enable us to securely accept help from others without fully trusting them.

In the corporate world, this type of design (minus the security/ownership part) is fairly standard. However, the tools are either proprietary or they are designed from a politically hierarchical perspective (rather than a single database server there is a cluster of database servers that still play a central role).

What would it mean to follow these goals in an web-application environment?

Authentication

Almost all web-applications have at least one piece of private data: every user’s password. In an ideal world, we would all use a different password on every web application, however, we all know that world doesn’t exist. Most people use the same password everywhere, making it an important piece of private information. For our purposes, we just can’t use passwords this way without the risk of people having them compromised.

Open ID, a distributed authentication system that allows you to specify a third party to authenticate you, provides an elegant solution to this problem. If our application used OpenID for authentication, then the web application itself would never need to know the actual user’s password.

However, I think there’s an even better solution. The oldest distributed authentication system is Pretty Good Privacy (PGP) - commonly known by the free implementation called Gnu Privacy Guard (gpg).

If our application used gpg for authentication, the process of getting an account would go something like this:

The user clicks the create account link
They are asked to enter the gpg public key and their gpg id (they are also provided a link to instructions on how to create a gpg key pair). We would only need to get the public key (not the gpg id), but by getting the gpg id as well, we’d be able to do things like check public key servers to see if a gpg id has been revoked.
We’d then ask them to verify that they have the private key by presenting encrypted text and ask them to decrypt it (software like the FireGPG plugin for Firefox would be critical to make this process user friendly enough).
Now they have an account. In the future, they can login by decrypting a random encrypted string and the server never has to know their password.

The primary added benefit of using gpg is: every time the user posts a blog, they can digitally sign the blog, so that if the blog is every altered by a malicious system administrator, the signature verification would fail making that alteration easy to detect.

Distribution

How would this application work on multiple servers in a coordinated way?

I’ll start by suggesting some terminology:

Spokes - Council design: the name I’m giving this approach. Technically, it’s still a Client - Server approach (because people will connect to it using their web browser clients), however, rather than connect to a single server, they will connect to a “council” of servers.
Council: refers to the collection of servers that are contributing to the particular implementation of the web application.
Spoke: refers to a single server in the council

Suppose I announce to the world: I’m starting a new blog server for the purposes of supporting a global, coalition-based organizing campaign. One aspect of the campaign is to get coalition members, most of whom have never blogged before, to blog about the campaign. Anyone can get a blog by going to blogs.example.org.

I have already setup a at least one spoke server (a.spoke.blogs.example.org), so when you go to blogs.example.org, you get a functional blogging application.

However, since I’m expecting a lot of growth (and I want to respect the three goals above) I also put a call out for tech collectives and organizations that are interested in supporting the project by donating server resources.

I am then contacted by Group B who says: We have a server connected to the Internet that we’d like to donate to the project.

I provide them with a script that they run on their server which downloads the source files, runs some tests to demonstrate that it is fully working, and then reports back to me that it is ready to go.

Then, I update the DNS record for blogs.example.org so that it will, round robin style, include the new server’s IP address when a user goes to blogs.example.org.

So far so good. Now what?

User Maria goes to blogs.mayfirst.org (and lands on b.spoke.blogs.mayfirst.org). She logs in with her gpg key and then clicks the button to create a blog. The server generates links in a form to ensure that she stays on the same server, although to Maria, it appears as though she is always on blogs.example.org.

She’s asked what username she wants for her blog and she selects “maria” (maria.blogs.example.org). She hits submit and recieves a message that her request is pending and asks her to check back in a few minutes.

Meanwhile, b.spoke.example.org executes a dns query (the oldest distributed database system in the world) to see if maria.blogs.example.org is taken. If not, it submits a request to the authoritative name server for blogs.example.org requesting that maria.blogs.example.org be created and given it’s own IP address. The authoritative DNS server could do any number of checks (complicated key exchanges or it could simply see if the IP address is an IP address belonging to a server in the DNS system), then it would create the record (provided nobody has slipped in earlier) and respond that the record was created.

Now, Maria can start blogging. When Maria publicizes her blog (http://maria.blogs.example.org), it is already setup to go to the right server, thanks to the domain name system.

Additional features could include: every user has a primary spoke plus one or more secondary spokes. All write requests are re-directed to the primary hosts, read requests are redirected to any of the secondary spokes. Perhaps when the original DNS record is created, it also creates edit.maria.blogs.example.org which points to the primary host, while maria.blogs.example.org is setup round-robin style to go to all of the secondary spokes. The secondary spokes are responsible for pulling in data from the primary spokes to stay in sync.

This approach is redundant: if one server goes down, a routine can be run on the name server to remove that IP address and, if the server is a primary server, then promote one of the secondary servers to take over as the primary server. It’s also scalable. Servers can be configured to refuse new bloggers if they start running low on hard disk space, or they could add additional secondary servers for popular bloggers if they run low performance-based system resources. Ideally there would be administrative scripts that could transfer blog accounts from server to server.

Aggregation and Indexing

There are still a few features that we would want that don’t scale very well. The whole purpose of building a giant blog network is to provide a sense of unity - we want all of the blogs to be aggregated and search-able to build this unity. No matter which server you land on, it should show you the most recent 20 blog posts from accross the entire network. You should also be able to type a search term in the search box and search all the blogs.

Aggregation and indexing would get significantly more difficult and resource intensive as we add more servers. Aggregating blogs on 5 or 10 servers is not so difficult, but doing that for hundreds of servers each of which have hundreds of blogs could become a monumental task. Although technically this is a scalability problem, given our model, it could be addresses by throwing more computers at it (as opposed to hitting a wall that can’t be overcome without hardware upgrade). We could organize concentric circles of aggregating servers, so each server is only aggregating 5 - 10 servers and then an upstream server aggregates their aggregate. The same model could be applied to indexing. These network could produce an inner circle of servers that contain the entire aggregate and index for all servers to access.

Unsolved Problems

How would we establish lines of trust. While the system works well if one untrusted spoke flakes out, what happens if several spokes (which encompass the primary and secondary servers for a particular blog) all flake out together?

Getting CPAN running locally

Tue, 08 Jan 2008 09:32:44 +0000

My goal was simple - to install the DateTime::Event::Recurrence CPAN module for my non-privileged user. As often is the case when I try to figure out CPAN (about once every two years - long enough to forget everything I knew about it previously) - it seemed a lot harder than it should have been.

I eventually succeeded by following these steps:

I made my own local perl directory:
```
 mkdir share
 mkdir share/perl
```
I exported a couple environmental variables. Note, by appending a colon to the end of MANPATH I’m instructing the manpath program to append the system default MANPATH paths to the end of my custom one (see man manpath). According to man perlrun, the PERL5LIB variable defines local directories to check before checking standard ones - so no need to worry about over-writing the standard library locations.
```
 echo "export PERL5LIB=~/share/perl/lib" >> ~/.bashrc
 echo "export MANPATH=~/share/perl/man:" >> ~/.bashrc
```
And then sourced them:
```
 source ~/.bashrc
```
I ran cpan with:
```
 cpan
```
I chose to let cpan auto configure my settings

From the cpan shell, I told cpan about my own directories:

 o conf makepl_arg "LIB=~/share/perl/lib \
 	INSTALLMAN1DIR=~/share/perl/man/man1 \
   INSTALLMAN3DIR=~/share/perl/man/man3 \
 	INSTALLSCRIPT=~/share/perl/bin \
 	INSTALLBIN=~/share/perl/bin \
 	PREFIX=~/share/perl"

 o conf makepl_arg "LIB=~/share/perl/lib \
   INSTALLMAN1DIR=~/share/perl/man/man1 \
 	INSTALLMAN3DIR=~/share/perl/man/man3 \
 	INSTALLSCRIPT=~/share/perl/bin \
 	INSTALLBIN=~/share/perl/bin \
 	PREFIX=~/share/perl"

Saved these settings for next time:
```
 o conf commit
```
I installed the latest version of CPAN with:
```
 install cpan
```
And then installed what I wanted:
```
 install DateTime::Event::Recurrence	
```

Getting Logitech camera working with Linux

Tue, 08 Jan 2008 09:32:44 +0000

Josue gave me a USB camera made by logitech - and I’ve been meaning to get it working with my Debian Lenny system for a while.

I started by plugging it in and watching syslog:

	Aug 14 14:13:08 liberace kernel: [128348.749001] usb 3-1: USB disconnect, address 4
	Aug 14 14:13:12 liberace kernel: [128352.657450] hub 4-0:1.0: unable to enumerate USB device on port 5
	Aug 14 14:13:12 liberace kernel: [128352.894212] usb 3-1: new full speed USB device using uhci_hcd and address 5
	Aug 14 14:13:13 liberace kernel: [128353.093083] usb 3-1: configuration #1 chosen from 1 choice
	Aug 14 14:13:13 liberace kernel: [128353.119601] usb 3-1: New USB device found, idVendor=046d, idProduct=08a9
	Aug 14 14:13:13 liberace kernel: [128353.119610] usb 3-1: New USB device strings: Mfr=0, Product=2, SerialNumber=0
	Aug 14 14:13:13 liberace kernel: [128353.119614] usb 3-1: Product: Camera

So far so good. I was hoping to see some reference to /dev/video, but that wasn’t part of the output and the device wasn’t created.

Next I tried:

	aptitude install qc-usb-utils

Still nothing. Then, I noticed that apt-cache show qc-usb-utils says: “These programs are completely useless without a qc-usb-modules package.”

There is no qc-usb-modules package (yes, Debian seems aware of that problem).

But there is a qc-usb-source package. I installed that package and then ran:

m-a

That’s module-assistant - which makes building a kernel module a snap.

After running it, I had to do some digging to learn that the module it built is called quickcam (I tried modprobe qc-usb to no avail). The following successfully loaded the module:

	0 jamie@liberace:linux$ sudo modprobe -v quickcam
	insmod /lib/modules/2.6.25-2-686/drivers/media/video/quickcam.ko 
	0 jamie@liberace:linux$

But, unplugging and re-plugging the camera produced the same output in syslog and no /dev/video.

More web searching finally turned up gspca. I removed the qc-usb modules and then:

	aptitude install  gspca-modules-2.6.25-2-686

Followed by:

	 modprobe -v gspca

And, to load in on my next boot:

	echo "gspca" >> /etc/modules

And now, miraculously, Ekiga recognizes and uses my camera!

Goals for a New Global Movement

Tue, 08 Jan 2008 09:32:44 +0000

The World Social Forum says: “Another world is possible.” How is this world shaping up? What role will the US play? What role will technology and the Internet play? How can we prepare?

It’s impossible to know what the world will look like in the future, but there are some unmistakable trends visible today. While the US has experienced recessions in the past, the current financial state of the country is bleaker than we’ve seen it in a long time. Meanwhile, China is economically growing and taking on super power stature in the world previously dominated by the United States. Increasingly, the most important role that US is making to the global economy is that of consumer, based largely on credit.

Similarly, the left in the US, once a significant force in the world, is increasingly overshadowed by revolutionary movements in Latin America, ranging from the World Social Forum movement itself (born in Brazil) to the Zapatistas novel creation of a local/world movement to leaders like Hugo Chavez re-writing US/Latin American relations.

These movements have had a powerful impact on the politics and culture of the region. For the first time in contemporary history, almost all the countries of Latin America elect their governments. Dictatorship, once the “norm” in the Spanish-speaking Americas, is not only not prominent but it has become a kind of atrocity in the popular political culture. Furthermore, the moves being made “from the top” (like Chavez in Venuzuala and Morales in Bolivia) have enormous and broad poliical support. In the case of Morales, his presidency was the culmination of years of grassroots mobilization and organizing.

There’s no mistake: the US is losing its central role in the world. Furthermore, the new world is increasingly based on international relationships and, from the left, the most powerful moves are being made in Latin America.

Without knowing anything else, these trends give us a lot to work with. For US activists, it means examining our own center-of-the-universe perspectives. Nobody is immune from the environments in which they are raised. How does the increasing irrelevence of the US change our political priorities? What does this mean for locally-focused projects when the policies and trends in our country are increasingly determined by other countries? How relevant are political projects that are not looking beyond our borders?

In many ways, former empire countries like the US and England are perfectly situated to build international movements. There’s a much higher percentage of bi-lingual and bi-cultural activists with close ties to other countries than any where else. However, growing up in an empire country has its draw backs as well. These global trends require US activists to adjust to being in a movement in which we’re no longer the center.

It also involves a change in scale. The left in the US is tiny compared to the global left. Our strategies, honed to work with memberships numbered in the hundreds or thousands, need to now accomodate tens of thousdands and hundreds of thousands.

While other countries may have more experience with scale, nobody has experience with the scale international movements can offer. How do we organize mass movements? Due to limits in communication, previous mass movements have often been based on top-down, undemocratic processes. Now, with remarkable advances in communications and the Internet, what options do we have to democratize these processes?

One unique asset that the US has is a relatively well developed liberal technology community. If nothing else, the nonprofit industrial complex has created enough opportunities for left-leaning technologits to make a living while developing extremely valuable Internet-related skills. It’s based on this community that projects like May First/People Link, Riseup (and in Canada, Tao and Resist) have been able to flourish.

As these projects have developed, a politics has developed with them as well, a politics that, while rooted in technology, has broad implications well beyond computers and the Internet.

Technologists, even liberal ones, have much more exposure to very radical ideas around openness, transparency, and freedom. These ideas come from some broad Internet-based movements (the free software movement being one of the most prominent) - but also from a deeper understanding of and experience with many technology experiments that are rooted in these ideas (like Wikipedia or the Debian Linux development team).

In addition, while the US still dominates most free software projects, the degree of internationalization in the free software world is astounding. Free software technologists have made an enormous contribution to understanding how we can build systems that operate transparently in multiple languages and with multiple cultures. This contribution is valuable in and of itself, but additionally gives US technologists, unlike our non-technological counterparts, some concrete experience in what it means to collaborate across borders.

These ideas, taken to their radical roots, are well suited for democratic, mass movements. These ideas, as they exist in the minds of technologists and non-technologists on the left today, however, are still very under-developed, particularly in the broader political context of building democratic mass movements.

How do we prepare and develop technologists for a more political role? How do we develop and prepare non-technologists for this role?

Politicizing technology

One strategy is to politicize our technology work. This strategy pertains as much to technologists as it does to non-technologists. One step in this direction is to consider common technology goals in a political light.

For example, consider these goals common to technology projects in a broader political context:

Redundancy: no person or server should ever occupy a role so central that if they were to be removed, the entire project would fall apart
Scaleability: we should start with a strategy that not only works with 50, but with 500 and 5,000 and even 5 million as well. Similarly, our strategy must scale culturally - it must function with people of all backgrounds, races, nationalities, social groups and sexes.
Transparency: all decisions, meeting notes, discussions should be in the open, publicy available for review. No information should be proprietary or horded.
Trust: new people should be able to join the campaign with enough responsibility to productively contribute and demonstrate their trustability, but without the ability to do great harm to the campaign.

How many of our political organizing campaigns adhere to these goals?

Certainly not all technology projects adhere to these goals. However, based on experience with the free software movement, many technologists do adhere to these goals and, in some political organizing campaigns, the technologists are the ones pressuring the political leaders on these goals.

These goals transcend technology - they shatter the distinction between the organizing and the tech work - demonstrating that building a global movement means that tech work is political work and must be full incorporated into the political thinking and development of the movement. Building from this model means that there are no non-techies: we are all techies of different skill levels, just as we are all organizers at different levels of political development.

These are the goals of a global movement, one that hasn’t been possible before now.

Got a tech question?

Tue, 08 Jan 2008 09:32:44 +0000

Great to see you all at the NYC Grassroots Media Conference. For those of you with tech questions that we weren’t able to answer, please post your question to our Support Page.

GPG for the masses

Tue, 08 Jan 2008 09:32:44 +0000

dkg and I spent the afternoon trying to figure out how to help people use GnuPG to send and receive encrypted email. We specifically were interested in programs that are cross-platform or run on a Macintosh (we have no Windows computers to test on). For linux users, Seahorse is hands down the best graphical user interface for GPG.

We tackled two issues:

Finding a decent GUI for managing keys.
Using a removable disk (USB thumb drive) to store GPG keys so they could be used on multiple computers.

Here are some of the notes.

GUIs for managing keys

GnuPrivacy Guard (GPA)

Our first hopeful, is a bit rough around the edges. It does run on both Windows and Linux, which is very promising. However, it’s not being updated in Debian or Ubuntu (last update was back in 2006). We installed the most recent upstream source (released in the summer of 2007) and compiled it.

The first problem happened when we tried to edit the key we had just generated. The only key editing options were to change the passphrase and the expiration date. There is no way to add a new user id to an existing key. Boo.

The second problem was when we tried to search for keys on a public key server. The search interface only allows for searching by key id (as opposed to user id

which would allow you to search for someone’s key by entering their email address).

A third problem is that, when signing a key, it forces you to sign all user ids on the key, rather than allowing you to pick the ones to sign. Furthermore, it doesn’t list all user attributes (a photo attribute on a test key we were using did not show up at all). And worst of all - it signs the photo attribute that is not displayed when you sign the key!

A fourth problem is that there doesn’t seem to be a way to revoke your own key or a key signature on another key.

Enigmail

Enigmail is a Thunderbird extension, with amazing documentation and a very well done user interface. We would have started with Enigmail, but it has the draw back of requiring Thunderbird. Enigmail does allow one to add a new email user id to a key (but no way to add a different user id, like a photo). Enigmail allows you to search public key servers for keys based on an email addresses.

On the signing side of things - if you right click an un-expanded key and select sign, Enigmail will sign all user ids. Worse, Enigmail allows you to expand a key to see the user ids, allowing you to right click on a single user id and select sign. Unfortunately, that operation also signs all keys.

Enigmail does allow you to revoke your own key, but no way to revoke a signature.

WinPT

WinPT could be good, but we don’t have a Windows machine to test it on. Furthermore, the last news item on the web site is a request for a new project lead (and is dated 2005-05-02). The last release seems to be dated 2003-04-23.

Gpg4win

Gpg4win provides a single package of a bunch of the gpg binaries for windows (including both WinPT and GPA), however, there are no new GUI key management programs offered.

MacGPG

MacGPG allows you to search by email address. However, you cannot add a user id from within the GUI (it opens up a terminal window with gpg running the -edit-key command). In addition, when trying to sign a key, MacGPG opens a terminal window prompting you to sign all user ids. When you click a user id and try to sign it, nothing happens.

Conclusions

Wow. We’re still a ways off here. All the packages do allow for basic key management operations (create a new key pair, verify a fingerprint, export a key). However, many fairly important functions are missing from all GUIs we looked at.

Using Removable Disk

We started with a Fat32 formatted USB key with the idea that it could be easily mounted on the most common operating systems in use today.

We ran into a failure when using gpg to create a private/public key pair. The failure happened when trying to hard link a temporary file (hard links are not supported on Fat32). This problem might not happen if we were using a gpg compiled for Windows (but we didn’t have a Windows machine to test with). We solved the problem by formatting the USB key with ntfs. Yuck.

Then, we ran into a problem trying to search for a key on a public key server - we received an error regarding file permissions of our gpg directory. Windows NT on Linux makes all files owned by root and world readable and writable. Double Yuck.

Grassroots Use of Technology Conference 2008

Tue, 08 Jan 2008 09:32:44 +0000

We had another amazing Internet Rights workshop

this one at the 2008 Grassroots Use of Technology Conference.

We had four groups of about four people each - a much smaller gathering than in the past. We also started the session 15 minutes late and old had an hour and 15 minutes to start with. Unfortunately we had to cut the discussion at the end.

On the positive side, we partnered with the Boston Action Tank. They ran the session immediately after ours. Taking the number one right (we ended up with two rights tied for first place - we chose the right to govern) we did a power analysis to determine what pre-conditions we need to achieve the right and who are our allies in the struggle. We didn’t have enough time to come out a publishable consensus - however, it was a useful and enlightening step to take with the rights.

And the rights are …

First Amendment Rights shall extend to all online communications. [4]
All users have the right to govern the Internet as a commons that allows participation and access for all. [4]
Keep the information as private as we want. Levels of privacy should be dictated by the users. [3]
Transparency about web site and network ownership. [3]
Universal availability of relevant tools and training for full participation in the digital environment [3]
Universal, free access to the Internet for everyone. [2]
Information sent from a “sending” machine should not be edited or obstructed in their transit to the intended “receiving” machine. [2]
All users have the right to form and self-govern online communities. [2]
Right to accountable name-anonymous access to the Internet. [2]
All users have the right to protect their data and transmissions from spying or editing. All users have the right to refuse to surrender their protection methods to individuals, organizations or governments. [1]

[The numbers in brackets are the number of groups that endorsed the right. 4 groups total.]

Ikiwiki + Git + Debian Etch

Tue, 08 Jan 2008 09:32:44 +0000

After a lot of head scratching, I seem to have ikiwiki up and running using a git repository - all running on Debian Etch.

I’m using the version of ikiwiki that ships with Etch (1.33.3), which doesn’t seem to have some of the nice features available in the current version (2.32.3) such as the ikiwiki-makerepo command. In addition, I am using git 1.4.4.4-2 (also on Etch).

Nevertheless, I was able to make it work. I did the following somewhat cumbersome steps. Note: if you are using a more recent version of ikiwiki, you probably shouldn’t follow these steps.

Creating the git repositories

Create a bare repository:

 mkdir iw-repo
 cd iw-repo
 git --bare init-db

Copy the sample blogs directory into a temporary directory
```
 cp -rp /usr/share/doc/ikiwiki/examples/blog/ temp
```
Initialize the temp directory as a git repository
```
 cd temp
 git init-db
```
Check in the files
```
 git add *
 git commit -a
```
Push to the bare repo
```
 git push ../iw-repo
```
Delete your temp directory
```
 cd ..
 rm -rf temp
```
Create at working directory from the bare repo
```
 git clone iw-repo iw-working
```

Setting up ikiwiki

Copy the sample setup file to your home directory

 cp /usr/share/doc/ikiwiki/html/ikiwiki.setup ~/

Edit the file. A big concept I missed the first time arround: the “wrappers” section refers to executable files that ikiwiki will create for you. There are two that you may be interested in:
- The ikiwiki.cgi wrapper is used by the web server to execute commands based on comments or posts added via the web browser.
- The post-update wrapper is executed every time you update the git repository - it triggers the wiki to be re-freshed.

The “wrapper” variable is the file location you want ikiwiki to create the wrapper file for you.

Setting up your local repository

Clone the repository:

 git clone user@host:/path/to/iw-repo local-ikiwiki

Now - make changes to any of the files.
Finally - commit your changes and push them upstream:
```
 git add
 git commit
 git push
```
And voila, it should be showing up on your site.

Installing asterisk

Tue, 08 Jan 2008 09:32:44 +0000

This is attempt number two. Attempt number one I didn’t properly document, so hopefully this attempt will be more successful just for that reason. Below is my final email on our previous attempt. The two differences with the current attempt are:

It’s June 2008 as opposed to September 2007. Hopefully we have new software that might have fixed problems from before.
New server hardware. We’re using a different computer.

Here’s the email of the previous problem:

	I've started going through and trying to debug. I'm seriously at the point of
	recommending that we remove the card and send it back for a replacement if not
	full refund (and go for a different phone system).

	I've limited experience with the zaptel stuff before and I find it incredibly
	flakey. I don't know if it's this particular card or this particular card
	model (it's a newer version). Several years ago I worked with a different
	zaptel card and, once we got it working, it seemed to work pretty
	consistently.

	In any event... this was my experience this evening:

	I confirmed the familiar problem of calling into the system and not having the
	system respond to key presses. And once again I confirmed that by shutting
	down asterisk, unloading and reloading the modules and starting asterisk
	again, the problem goes away (for the time being).

	I created a script in /root called reset-asterisk that will take care of this
	in one fell swoop.

	I tested dialing out from both phones on 718, 917 and did not get the "You
	must dial a one" error. However, as I continued to test I did get the error
	and believe that it is intermittent. Sigh. Looking at the console, asterisk
	appears to be sending out the one with the number - maybe it's not waiting
	long enough for the dial tone so the one is not properly going through?

	I also found that sometimes I could pick up both internal phones and make two
	simoultaneous calls (at least once) but most times I would get that error or a
	busy signal.

	To make matters worse, at two points in my testing the entire server froze up
	(did not respond to pings) forcing me to hard reset the machine.

	My sense is that most if not all of these problems are related to the zaptel
	card and/or kernel modules. And what makes this so difficult is that all the
	problems seem to be intermitten!!

	I'm not sure what to do at this point, but am open to suggestions!

	jamie

	p.s. I made the following changes on the server:

	* Enabled serial console access (modified /etc/inittab and
	+/etc/boot/grub/menu.1st)

	* Uploaded "The Man in Black.mp3" into the /var/lib/asterisk/moh directory. On
	one server crash, asterisk was trying to access that directory (moh = music on
	hold). I'm not sure why, but thought I would throw a file into that directory
	to see if that was why it was failing.

	* changed extensions.conf to use a different programming logic to select the
	line to dial out with (based on:
	http://www.voip-info.org/wiki/index.php?page=Asterisk+cmd+ChanIsAvail).

We’re starting with a Debian Lenny installation that is stripped down to run just the basics. Then:

	aptitude install asterisk
	aptitude install zaptel zaptel-source

Then, based on /usr/share/doc/zaptel/README.Debian.gz, I ran:

	module-assistant a-i zaptel

Woops - got an error:

	Bad luck, The kernel headers for the target kenrel version could not be
	found and you did not specify other valid kernel headers to use.

Tried again with:

	module-assistant prepare

And got:

	The following packages have unmet dependencies: linux-headers-2.6.22-3-686:
	Depends: linux-kbuild-2.6.22 but it is not installable E: Broken packages

Strange. Linux 2.6.24-1-686 is installed:

	0 zaius:/usr/share/doc/zaptel# aptitude search linux-image | grep ^i
	i   linux-image-2.6-686             - Linux 2.6 image on PPro/Celeron/PII/PIII/P
	i A linux-image-2.6.22-3-686        - Linux 2.6.22 image on PPro/Celeron/PII/PII
	i A linux-image-2.6.24-1-686        - Linux 2.6.24 image on PPro/Celeron/PII/PII
	0 zaius:/usr/share/doc/zaptel#

But is wasn’t chosen when I rebooted:

	0 zaius:/usr/share/doc/zaptel# uname -a
	Linux zaius 2.6.22-3-686 #1 SMP Sun Feb 10 20:20:49 UTC 2008 i686 GNU/Linux
	0 zaius:/usr/share/doc/zaptel#

I ran update-grub:

	0 zaius:/usr/share/doc/zaptel# update-grub 
	Searching for GRUB installation directory ... found: /boot/grub
	Searching for default file ... found: /boot/grub/default
	Testing for an existing GRUB menu.lst file ... found: /boot/grub/menu.lst
	Searching for splash image ... none found, skipping ...
	Found kernel: /vmlinuz-2.6.24-1-686
	Found kernel: /vmlinuz-2.6.22-3-686
	Updating /boot/grub/menu.lst ... done
	0 zaius:/usr/share/doc/zaptel#

Checked /boot/grub/menu.lst and made sure default was set to 0 and the 2.6.24 kernel was the first listed and then rebooted.

Now we are using 2.6.24:

	0 zaius:~# uname -a
	Linux zaius 2.6.24-1-686 #1 SMP Thu May 8 02:16:39 UTC 2008 i686 GNU/Linux
	0 zaius:~#

Strange. Not sure what happened before.

Now running module-assistant a-i zaptel returns without an error (it installed cpp-4.1 gcc-4.1 gcc-4.1-base libmudflap0 libmudflap0-dev linux-headers-2.6.24-1-common linux-kbuild-2.6.24) and then built the kernel modules from the zaptel-source.

Now:

	0 zaius:~# lsmod | grep zap
	1 zaius:~# modprobe zaptel
	0 zaius:~# lsmod | grep zap
	zaptel                188548  0 
	crc_ccitt               2176  1 zaptel
	0 zaius:~#

Ok. So far so good. The zaptel device seems created:

	0 zaius:~# find /dev/zap
	/dev/zap
	/dev/zap/ctl
	/dev/zap/pseudo
	/dev/zap/channel
	/dev/zap/timer
	0 zaius:~#

Continuing with README.Debian.gz:

	/etc/zaptel.conf
	----------------
	A sample /etc/zaptel.conf is no longer installed by default. You should 
	generate it manually (or automatically with genzaptelconf) if and when
	you actually have zaptel hardware and installed a zaptel-modules package
	for your kernel version.

Ok.

	0 zaius:/usr/share/doc/zaptel# genzaptelconf
	0 zaius:/usr/share/doc/zaptel#

That was really anti-climatic. Nonetheless, it apparently worked:

	0 zaius:/usr/share/doc/zaptel# cat /etc/zaptel.conf 
	# Autogenerated by /usr/sbin/genzaptelconf -- do not hand edit
	# Zaptel Configuration File
	#
	# This file is parsed by the Zaptel Configurator, ztcfg
	#

	# It must be in the module loading order


	# Global data

	loadzone	= us
	defaultzone	= us

	0 zaius:/usr/share/doc/zaptel# cat /etc/asterisk/zapata-channels.conf 
	; Autogenerated by /usr/sbin/genzaptelconf -- do not hand edit
	; Zaptel Channels Configurations (zapata.conf)
	;
	; This is not intended to be a complete zapata.conf. Rather, it is intended 
	; to be #include-d by /etc/zapata.conf that will include the global settings
	;
	0 zaius:/usr/share/doc/zaptel#

Hm. I think it’s supposed to list our FXO/FXS modules in /etc/asterisk/zapata-chanels.conf. From our previous install, we have 6 channels:

	0 zaius:~# cat asterisk.from.blackbolt/zapata-channels.conf
	; Autogenerated by /usr/sbin/genzaptelconf -- do not hand edit
	; Zaptel Channels Configurations (zapata.conf)
	;
	; This is not intended to be a complete zapata.conf. Rather, it is intended 
	; to be #include-d by /etc/zapata.conf that will include the global settings
	;

	; Span 1: WCTDM/0 "Wildcard TDM800P Board 1" 
	;;; line="1 WCTDM/0/0 FXOLS"
	signalling=fxo_ls
	callerid="Channel 1" <6001>
	mailbox=6001
	group=5
	context=from-internal
	channel => 1
	callerid=
	mailbox=
	group=
	context=default

	;;; line="2 WCTDM/0/1 FXOLS"
	signalling=fxo_ls
	callerid="Channel 2" <6002>
	mailbox=6002
	group=5
	context=from-internal
	channel => 2
	callerid=
	mailbox=
	group=
	context=default

	;;; line="3 WCTDM/0/2 FXOLS"
	signalling=fxo_ls
	callerid="Channel 3" <6003>
	mailbox=6003
	group=5
	context=from-internal
	channel => 3
	callerid=
	mailbox=
	group=
	context=default

	;;; line="4 WCTDM/0/3 FXOLS"
	signalling=fxo_ls
	callerid="Channel 4" <6004>
	mailbox=6004
	group=5
	context=from-internal
	channel => 4
	callerid=
	mailbox=
	group=
	context=default

	;;; line="5 WCTDM/0/4 FXSKS"
	signalling=fxs_ks
	callerid=asreceived
	group=0
	context=from-pstn
	channel => 5
	context=default

	;;; line="6 WCTDM/0/5 FXSKS"
	signalling=fxs_ks
	callerid=asreceived
	group=0
	context=from-pstn
	channel => 6
	context=default

	0 zaius:~#

Trying again with -v (verbose) and -d (hardware detection):

	0 zaius:~/asterisk.from.blackbolt# genzaptelconf -vd
	Unloading zaptel modules:

	Temporarily moving zaptel.conf aside to work around broken modprobe.conf
	Test Loading modules:
		- 	wct4xxp	
		- 	wcte12xp	
		- 	wcte11xp	
		- 	wct1xxp	
		- 	wanpipe	
		- 	tor2	
		- 	torisa	
		- 	qozap	
		- 	vzaphfc	
		- 	zaphfc	
		- 	ztgsm	
		ok	wctdm24xxp	
		- 	wctdm	
		- 	opvxa1200	
		- 	wcfxo	
		- 	pciradio	
		- 	wcusb	
		- 	xpp_usb	
	Generating '/etc/zaptel.conf and /etc/asterisk/zapata-channels.conf'
	Note: generated /etc/asterisk/zapata-channels.conf not included in zapata.conf
	To fix:  echo '#include zapata-channels.conf' >>/etc/asterisk/zapata.conf
	Reconfiguring identified channels

	Zaptel Version: 1.4.10.1
	Echo Canceller: MG2
	Configuration
	======================


	Channel map:

	Channel 05: FXS Kewlstart (Default) (Slaves: 05)
	Channel 06: FXS Kewlstart (Default) (Slaves: 06)

	2 channels to configure.

	0 zaius:~/asterisk.from.blackbolt#

Hm. Now it found 2 our of the 6 channels. That’s an improvement. The channels it found are the FXS channels (the ones you plug a telephone line into, as opposed to FXO which plug a telephone into).

Ok. Enough for tonight…

Internet Part 1 and Part 2

Tue, 08 Jan 2008 09:32:44 +0000

To use the People’s Production House slogan: The Internet is Yours … If you want it.

PPH started Part 1 of the workshop by exploring the Digital Expansion Initiative. They shared both the results and the methods they use to learn about how people use the Internet and how we want it to develop (see some of their testimony).

I really appreciated their transparency: we went through the various methods for collecting research (via interviews, drawing pictures, etc) - for each method we all did it and discussed the effectiveness of the strategy.

Part II was May First/People Link - and the Internet Rights workshop.

And the results are in. The number in brackets is the number of groups that endorsed the right. We had 5 groups total.

Free Speech and peacable assembly without permit for anyone, everywhere. A right including the right to dialogue. [4]
Right to internet and technology public education. [4]
All people have the right to be involved in a public global democratic process to manage the full spectrum of communications resources: [3]
Right to transparency about government and corporate activity, surveillance, activities. [2]
Right to personal privacy; Control and consent over your personal information; transparency of information collection practices. [2]
Airwaves/spectrum are a public resource controlled and managed by the people for public, not private, gain. [2]
Community ownership of internet infrastructure (domain name system, hardware, last mile connection, software source, bandwidth) [1]
Network managers and/or Internet Service Providers (ISPs) cannot discriminate against content of any type. [1]
all people have the right to contribute content; Rights of the common over intellectual property rights [1]
Peoples throughout the world have free access to ubiquitous broadband and computing resources through public institutions such as schools and libraries or any local resource center if they want it. [0]

Internet Rights Workshops the NYC Grassroots Media Conference

Tue, 08 Jan 2008 09:32:44 +0000

The results are in.

On March 2, 2008, May First/People Link organized a session to collaboratively develop a series of Internet Rights at the Grassroots Media Conference.

We ran the same workshop at the 2007 US Social Forum with different, but similar results.

At the GMC, we had less time, which seems to be reflected in the more raw wording. Also of note - none of the rights were endorsed by all groups (the number of endorsers are in brackets). We had a total of seven groups. And, of course, due to the nature of the project, there was some last minute back and forth - as you might imagine the last right was rather contentious!

Freedom of expression. [5]
The right to space, hardware, software, and non-restricted use of existing and future internet technology, including the right to not use a technology. [4]
Labor rights for internet workers and technology produced with priorities of ecological sustainability, labor justice and respect of community land (production and disposal). [3]
The right to a participatory governance process of the internet, including those who are not yet online. [3]
Free, equitable, and open access to the internet. As well as the codified right to not participate. [3]
Moderation against libel, slander, and defamation through the right to rebuttal. [2]
Right to privacy, and anonymity in all network based activity. [2]
The right to a domain name that is short descriptive and memorable, including equal-opportunity indexing. Non-fee-based promotion and searchability. Freedom from commercialization and speculation. [1]
Internet service provided by multiple, independent providers who compete vigorously and offer access to the entire internet over a broadband connection, with freedom to attach within the home any legal device to the net connection and run any legal application. [1]
The right for communities to enforce standards/values via censorship. [1]

It's called struggle for a reason

Tue, 08 Jan 2008 09:32:44 +0000

The NY Times ran an article last weekend about just how darn difficult it is to be an environmentalist. You switch from cardboard milk cartoons to recycled glass bottles to reduce waste and then, before you know it, somebody is telling you that all that glass recycling wastes energy. Sheesh!

The article is a nice illustration of the obstacles to building a political movement around a topic that has become marketable. Once environmentalism became an option at mainstream stores, something significant changed. We stopped seeing environmentalism as a complex problem requiring lots of discussion and, more often than not, messy solutions requiring difficult compromises. Suddenly, environmentalism became easy choice: buy product A instead of product B.

The good news is that a concept of environmentalism reached a huge new audience because it was made easy. The bad news is that reaching a new audience didn’t solve the underlying problems. Additionally, the transformative impact of engaging in environmental struggle, the most important and difficult aspect of any political movement, was lost. Now the environmental justice movement has to fight both environmental destruction and a mainstream perception of what environmentalism is.

Technology is on a similar trajectory. How do we engage people in a difficult discussion about the politics of our work on the Internet when even the left sees technology as choosing product A versus product B?

Consider some of the most pressing political issues facing the Internet: using free software and using OpenPGP and the web of trust to sign and encrypt our messages.

Making a commitment to free software is hard. It’s not just hard because you may need to learn how to use a new program. It’s hard because you have a new relationship to software development - one that has higher expectations of you. One that expects you to engage with the developers by posting bug reports, upgrading so developers can spend less time supporting old versions, and contributing answers to other users. It’s much harder for developers as well - demanding a much more engaged relationship with potentially thousands of users. The transformative potential of free software isn’t just to change the licenses of the software we use, but to transform the relationships of the developers and users of the software - from a capitalist exchange to a real community of collaboration.

Using OpenPGP to encrypt and sign our messages does make our lives more difficult. Not only because you have to learn new buttons to push in order to read and write email, but because good security requires us to fundamentally understand what we are doing. Security culture demands both adherence to protocol and, more importantly, a fundamental understanding of the protocol.

The mainstream (x509) protocol for security is significantly easier to understand: here’s a list of entities (in practice mostly corporations) that you should trust because we tell you to. Easy. You just trust them. The OpenPGP model allows you to build a network of trusted individuals. Much more complicated.

These things don’t come easy. They never will. The real question is: why do we expect them to be easy?

Keybindings and Openbox

Tue, 08 Jan 2008 09:32:44 +0000

I just had a terrifying moment when none of my custom key bindings worked. I was completely disabled. I use Openbox as my window manager, which is a great lightweight window manager with excellent support for custom keybindings. Except when they don’t work.

Part of my problem is that I was running xorg packages from unstable in an effort to trouble shoot my laptop crashing when I put it to sleep. After several months, I began to think that the problem was due to MySQL (now I shutdown MySQL and pause for 4 seconds before going to sleep). So, I finally decided it was time to downgrade my xorg packages to Lenny (testing). After restarting Openbox, many of my keybindings stopped working.

Fortunately, I learned a lot in the process of trouble shooting.

I soon discovered that my problems were all related to my use of the “windows” key - none of my key bindings that used that key worked. Everything else worked fine.

I started by backing up my openbox rc.xml file and starting a new one. I noticed that in the new one, the key binding for the windows key was “W” whereas I was using “w”. I switched to capital W but still no luck.

In my research I learned about the xev command. It’s really cool. It prints out what key your computer heard you press.

When I pressed the windows key the output was:

KeyPress event, serial 32, synthetic NO, window 0x2200001,
root 0x57, subw 0x0, time 1040577393, (134,99), root:(1122,601),
state 0x0, keycode 115 (keysym 0x0, NoSymbol), same_screen YES,
XLookupString gives 0 bytes: 
XmbLookupString gives 0 bytes: 
XFilterEvent returns: False

KeyRelease event, serial 32, synthetic NO, window 0x2200001,
root 0x57, subw 0x0, time 1040577525, (134,99), root:(1122,601),
state 0x0, keycode 115 (keysym 0x0, NoSymbol), same_screen YES,
XLookupString gives 0 bytes: 
XFilterEvent returns: False

From this output, I learned that they key in question had keycode 115 and that it reported NoSymbol. I learned that this should read Super_L.

I fixed it by running:

	xmodmap -e 'keycode 115 = Super_L'

However, Openbox still didn’t behave.

Then, I read the Openbox upgrade notes relating to the Windows key binding and learned that:

	Key bindings that use W as a modifier key have changed. W used to be
	hard-coded to be Mod4Mask. Now it has been changed to whatever mask you have
	the Super mod key bound to.

	You can use xmodmap to see your current bindings. Most modern distributions
	bind the windows key to Super_L, and bind Super_L to Mod4Mask. So this should
	probably not affect you unless you have a highly customized system, or have
	customized your xmodmap.

Then I ran:

	0 jamie@liberace:~$ xmodmap -pm
	xmodmap:  up to 2 keys per modifier, (keycodes in parentheses):

	shift       Shift_L (0x32),  Shift_R (0x3e)
	lock        Caps_Lock (0x42)
	control     Control_L (0x25),  Control_R (0x6d)
	mod1        Alt_L (0x40),  Alt_R (0x71)
	mod2        Num_Lock (0x4d)
	mod3      
	mod4        
	mod5        Scroll_Lock (0x4e)

	0 jamie@liberace:~$

It appears as thought mod4 is not set to Super_L, so I ran:

	xmodmap -e 'add mod4 = Super_L'

And now everything is working. I added the two xmodmap commands to my .xsession so they are started every time I log in.

I’m still not sure what happened - but I think my switch from unstable and then back to testing caused something to happen to my key mappings. Maybe when the xorg packages I was using make it into testing I will no longer need those lines in my .xsession file.

Khalil Gibran International Academy

Tue, 08 Jan 2008 09:32:44 +0000

Thanks for the work of May First/People Link member AWAAM, I’ve been following the story of Khalil Gibran International Academy, a school that opened in 2007 in Brooklyn to serve as a dual-language Arabic/English school where a mixed group of Arabic speakers and non-Arabic speakers would learn together.

The story of the school’s opening and the original principal getting pushed out because she acknowledged that “intifada” was a word with meaning and history is tragic. However, that was just the beginning. Seth Wessler has written an article documenting the continued impact of a conservative campaign to undermine the school with the latest twist: pictures of mosques being cut out of the text books.

Lessons in Collaboration: nytimes-se.com

Tue, 08 Jan 2008 09:32:44 +0000

On Wednesday, November 12, a collaboration of activists and pranksters distributed a remarkably high quality spoofed copy of the New York Times along with an equally artful website.

The reaction was phenomenal. News of the project was forwarded, blogged, discussed, txt’ed; subjected to rants and raves; and picked up by news organizations all over the world.

The server hosting the website, run by May First/People Link members The Yes Men got slammed. Almost immediately. And, was practically unreachable for the first day.

By the next day, the website was being distributed over four servers in three locations (later extended to six servers in six locations) and was being supported by activist techies spanning close to a half dozen radical tech collectives throughout North America, including May First/People Link, Riseup, Indymedia, Koumbit, Guerillartivism and more. And, the site was delivering page views faster than anyone expected.

What happened? How was this turn-around possible?

Day 0

Before this idea was even born, The Yes Men chose to host with a politically progressive provider, not a corporate provider. This step is one of the most crucial steps any politically focused organization can choose. Here are a few reasons why:

Activist providers have more resources. This assertion sounds counter-intuitive to the way we think about the left and capitalism - we’re used to being in situations where the left is under-resourced and the capitalists are awash in venture capital funding. However, when it comes to the Internet, hardware/capital is only one of many resources we need. Labor, particularly highly skilled labor available on a moment’s notice is far more critical. And activists providers are impressively organized with high caliber skills. Furthermore, the hardware costs are increasingly coming down to a point where even under-financed activist groups can afford them. And finally, in case you haven’t noticed, venture capital is waning these days. Really. Waning.
Activist providers are more flexible. We’re used to calling our corporate service providers and getting someone on the phone relatively quickly. That’s great. Except the person we get on the phone usually knows less than we do and has no power or authority to do anything. Activist providers don’t have the resources to be on call 24/7, but once we got on a problem, we have the flexibility, authority, and knowledge to help. Furthermore, we have the ability to call in our networks to bring in resources beyond our organization.
Activist providers actually want you to get more traffic so that you’ll win. We experience a lot of strange hand-wringing from our members over what happens if we get a lot of traffic. That’s understandable since commercial providers want you to get more traffic so that they can charge you more. Activist providers want you to get more traffic so that you’ll win. Most of the activist providers who stepped in to support the Yes Men will be getting higher than usual bandwidth bills this month. Although we’re all strapped for cash, this is a good thing and the very reason why we exist.
Activist providers extend the organizing project to the Internet. By choosing an activist provider, the Yes Men made a conscious and active decision to share their success with the radical tech movement. We’re stronger as a result, meaning the next time we have a similar situation with another site or another group, we will be that much more prepared and ready thanks to this decision by the Yes Men.

By hosting with an activist provider, the project had a different level of access to the network of people and organizations that eventually made the site sing.

Day 1

The site, which is running a free content management system called WordPress, is on a mostly dedicated, very powerful server (4 processors, 4 GB of RAM). Nonetheless, by the middle of day one the server was on it’s knees.

When the server went into overload, we immediately created an Internet chat room open to the public to help figure out how to get the site up and running. Although the handful of people working on the server were able to make small improvements, the real change happened when we were joined by our allies from Riseup and the network of tech activists they had access to. May First/People link has at our disposal two dozen machines in 5 locations around the country and an impressively skilled tech team. When we were joined by our allies, the resources at our disposable (directly and indirectly by our ability to grow even larger) became incalculable.

We spent the next several hours setting up caching servers around the country to reduce the load on the primary server. We experienced a lot of serious technical hurdles in the process, however, between the half dozen techies involved, we were able to declare success at 1:30 am. We experienced a few blips and minor problems, however, for the most part, the site was being successfully server to everyone who came.

Day 2

The next day we all monitored the site and the caching servers, eventually increasing the number of servers involved roe 4 to 6. We experienced a few minor problems, however, the site worked so well that conversation on the chat channel turned to brain storming new content to add.

Lessons Learned

“If we could do it over again” is a common refrain on all projects and this one is no exception.

It is very difficult to predict success and even harder to predict server load of a successful project, since every year the numbers of people who might view our site changes along with the software and hardware we use to power the site. At May First/People Link we have members predicting huge spikes that never materialize, while at the same time experiencing spikes they had no idea were coming. Nonetheless, with more advance warning, we could have had the caching network in place for the day of the launch. Additionally, we should have put a call out to our network for help immediately, which may have resulted in the caching system setup happening earlier on the first day.

For obvious reasons, the project chose to use the commercial DNS provider joker.com to host their domain name so that if anyone queried who was handling their domain name, the query would return:

a.ns.joker.com b.ns.joker.com c.ns.joker.com d.ns.joker.com

Yes, irresistible.

However, since the domain name was under the control of a corporation, not one of our allies, we were unable to properly control how long an IP address was assigned to the domain name nytimes-se.com. This lack of fine grained control made if difficult to switch to the caching system we had in place, and made it difficult to remove mis-configured caching servers.

Technical details

For the technically curious, below is a brief synopsis of what we did.

Our first move was to install and enable the WordPress supercache. For a while we thought it was broken because we experienced a lot of Redirect errors:

Request exceeded the limit of 10 internal redirects due to probable
configuration error. Use 'LimitInternalRecursion' to increase the limit if
necessary. Use 'LogLevel debug' to get a backtrace., referer:
http://www.nytimes-se.com/nytse/wp-content/themes/nytimes/style.css

However, we later realized that the errors were caused by an errant .htaccess file one directory up from the root directory.

Next we worked on getting the right number of MaxClients set in the Apache configuration. Too many clients and the server load sky rocketed. Too few clients and the server would start refusing connections. After a lot of back and forth we settled on 256.

Also, the following sysctl parameters were changed from 60 to 15 and from 7200 to 1800 respectively:

/proc/sys/net/ipv4/tcp_fin_timeout
/proc/sys/net/ipv4/tcp_keepalive_time

Next we tried to off load some of the large files (a few pdfs and mov files). After struggling for too long trying to get a fancy ModRewrite to work that would have allowed load balancing between multiple servers, we just put in a RedirectTemp to a single server.

Finally, we moved the database driving the WordPress site to another server.

For a few extra CPU cycles, we temporarily turned off two services: munin, and cron.

Despite all of these efforts, we were hitting loads of up to 170 and nobody could access the site.

We next worked on what appeared to be some WordPress rewrite rules going awry, which may have been contributing to the load problems (loops and loops and loops). At the same time we started setting up Squid proxy servers to help balance the load. Between the pool of techies working on the project, we got four squid servers up and running very quickly.

The next period was the most frustrating. It was really hard to trouble shoot the rewrite/looping rules after we had the proxy system in place - we couldn’t tell if the looping was caused by the proxy setup or was an undetected error prior to moving to the proxy setup.

Eventually, we stopped the loops with a one line WordPress plugin.

However, Squid and WordPress still gave us problems. They simply do not seem to get along very well (specifically - Squid re-writes http requests that is passes back to WordPress in a way that WordPress cannot handle). After hours and hours of trouble shooting, a brilliant 1:00 am suggestion was made: let’s switch to varnish (an alternative caching server). Varnish was setup in a matter of minutes (much simpler to configure) and worked extremely well. In the end, we got three varnish servers up and running.

The stats

A look at the stats is quite sobering. Normally, in our colo center in Telehouse, our combined membership uses just under 10 Mbits. Below is the graph for just the nytimes-se.com server:

[[images/nytse.png]]

And here are statistics for just one of the caching servers:

[[images/cache.png]]

The offline world

Fortunately, thanks to some members of the Rude Mechanical Orchestra (and many many others) there were other methods for people to access the paper:

[[images/paperboys.jpg]]

Look for us in Boston at the Grassroots Use Of Technology Conference

Tue, 08 Jan 2008 09:32:44 +0000

The Grassroots Use of Technology Conference is coming up!

May First/People Link members will be there in force. Ross, Agaric Design, Alfredo and yours truly will be teaming up to present the Internet Rights Workshop at the 1:30 pm breakout session.

Even more exciting - we’re partnering with the Boston Action Tank. They’re session will happen directly after ours and will pick up where ours leaves off. Be sure to go to their 3:00 pm session (Strategies for shaping the media/tech future) - to figure out how we will realize the Internet Rights we come up with in the earlier session.

Media Sanctuary on the future of censorship

Tue, 08 Jan 2008 09:32:44 +0000

Sunday morning at 10:00 am in Room J you’ll find me at Steve Pierce’s workshop on censorship.

UPDATE: I can’t believe I missed this workshop :(. Too many amazing workshops at the same time!

Be sure to checkout the Media Sanctuary and the story about Wafaa Bilal’s “Virtual Jihadi”.

NYC Grassroots Media Conference

Tue, 08 Jan 2008 09:32:44 +0000

The NYC Grassroots Media Alliance has announced the date for the next conference: Sunday, March 2 - be sure to mark your calendars. More information, including workshop proposal guidelines are available on their website: http://nycgrassrootsmedia.org/.

OpenID for content

Tue, 08 Jan 2008 09:32:44 +0000

Every day we post content on web sites run by people that we trust, or know a little, or don’t know at all, or in many cases explicitly don’t trust, or really, just about every variation within these categories of trust. Even with people we know and trust, we often have no idea how secure their systems are or how reliable their backups are.

OpenID directly addresses this concern, by asking: why would you entrust your password and identity to all of these websites run by people that we don’t know? With OpenID, the answer is: you shouldn’t! Instead, pick a single trusted provider to store your password and verify your identity. Then, any web site, trusted or not, directs you to your trusted identity provider to verify your identity and, provided the response is positive, lets you in without every needing to touch your password.

Then, satisfied and feeling secure, we proceed to pour hours and hours into writing posts, comments, how-to’s; uploading music, artwork, and and photos; and engaging in critical dialogues all of which could disappear permanently with the flip of one switch, the exploit of a single vulnerability, the sale of a corporation, or the disappearance of a collective.

What if we had a different model for content on the web?

What if, like with OpenID, we chose a trusted content publisher and published all of our content there? What if web applications allowed you the option: publish your blog, comment, photo, etc. on our servers - or publish on your open content enabled provider and leave a link on our server? The web application would display the content exactly the same - whether it was stored on their server or stored on your trusted provider’s server.

This setup in no way ensures that an untrusted web site won’t change your content before displaying it, or delete the post by deleting the link back to you, or do any number of devious and evil things. However, they won’t be able to delete or change the actual content that you wrote. The content is under your control.

A somewhat obvious criticism comes to mind: this approach means that if your trusted provider has a disaster, you could lose everything you’ve ever published on the Internet. Well, yes, that’s a problem. With the current paradigm, if a web site goes down, you’ve lost everything you’ve contributed to that web site, but not everything you’ve ever contributed anywhere.

On the other hand - wouldn’t you like a system that could allow you to download to your own computer everything you’ve written online with the click of a single button? While you are increasing the liklihood that a single mess-up could do very significant damage, you are at the same time giving yourself a degree of control over your data that is impossible in today. While your provider should certainly make their own backups, you could easily make an extra backup whenever you wanted. And, with the ability to easily backup your own data, you could also easily move to any provider you want at any time.

After re-reading what I’ve written - this idea seems really un-original. Isn’t this what http was designed to do?

Optimizing a drupal site in a hurry

Tue, 08 Jan 2008 09:32:44 +0000

To help boost performance on the http://wsf2008.net site Ben asked me to research APC and memcache. Here are my initial notes.

First - the relevant web sites:

Memcache home page: http://www.danga.com/memcached/ Memcache drupal module: http://drupal.org/project/memcache Memcache php manual http://us2.php.net/memcache

APC home page: http://pecl.php.net/package/APC APC drupal module: http://drupal.org/project/apc APC php manual: http://us2.php.net/apc

Both packages are designed to speed up programs by providing a faster cache for storing and retrieving data (faster than say, storing it in a database or in a php file-based session, etc.).

Memcache is a generic cache - there are api’s written for perl, python, php, etc. It’s available in debian as php5-memcache. APC, on the other hand, is specifically designed for PHP and is distributed via pecl/pear (pecl install apc).

From a drupal perspective - both modules seem to focus on replacing the caching function - which by default write to the database. Therefore - this is a good solution if the database is your bottleneck.

Riseup and Texas MEP on Social Networking

Tue, 08 Jan 2008 09:32:44 +0000

Riseup and Texas MEP put together a workshop on social network from a critical perspective. It was interesting to watch the tensions within the presentations. Texas MEP were full-on Facebook/My Space/etc. users. While being conscious of the risks, their position was that they are careful to only put on information that is public.

Brenna from Riseup demo’s Riseup’s installation of crabgrass - a social networking site designed from a collective/organizing perspective rather than an individuated perspective. She provided much of the critical analysis of the risks involved with corporate social networking sites (lack of privacy, reliability problems, etc.).

Brenna also provided Riseup’s 5 horseman of the privacy apocalypse:

relational surveillance: analysis of social networks via email and phone transactions (by the government) relational-surveillance
data profiling: the aggregation of consumer data in order to build detailed profiles on the consumption habits of everyone. data-profiling
tethered computing: devices that are controlled via a ‘tether’ by the manufacturer. On the desktop, trusted computing can be seen as a way of achieving tethered computing on an otherwise agnostic and innovative device. (by corporations and the government). tethered-computing
Geo spacial surveillance: location tracking via RFID, cell phones, IP addresses (by corporations) Geo spatial-surveillance
biometric surveillance: biometric scanning via CCTV face recognition, DNA databases. biometric-surveillance

That alone made the workshop worth it. I think we struggle a lot to figure out how to communicate security concerns. The organization of these concerns - specifically the way these 5 issues are abstracted from the specific applications - is really helpful.

We had some good discussion - one person mentioned how she’s uncomfortable with publishing our networks on corporate run servers.

The parting words of the workshop: We’re not just fighting to get our media out, but fighting to build and own the infrastructure.

Setting Bell Beeps in SILC

Tue, 08 Jan 2008 09:32:44 +0000

In the best tradition of “quick write it down before I forget it again” I am blogging about how to enable two people running silc to send each other a beep. Both users should be running Linux and the silc terminal client. And, both users should have beeping configured on their computer (you can test by opening a terminal window and typing Ctrl-g and you should hear a beep).

Then - in silc, turn bell_beeps on with:

/set bell_beeps ON

Now, if you are both in the same silc room, typing: CTRL-g followed by pressing the enter key should cause your computer to beep and the computer of everyone in the silc room (provided their silc configuration has bell_beeps set to ON as well) to beep as well.

Although I’m a big hater of beeping, I find this setup really useful because it allows me to to have a silc window open and buried or otherwise not visible, yet still allow people to get my attention.

SMTP Problems, old and new

Tue, 08 Jan 2008 09:32:44 +0000

I received 100 back scatter messages in my inbox today. As someone who receives mail for dozens of system email messages (like root@ and www-data@), this number is higher than usual, but not something so odd that it concerns me. Upon closer examination, however, I noticed two things different with these messages.

Normally, I get back scatter of two varieties: spammer emails joe@example.org. Joe is a legit user with a auto responder - so example.org accepts the spam, and then generates the auto response, which I receive. The second variety: spammer emails announce@lists.example.org. Since announce is a legit list, it is accepted by the mail server, handed off to the list server program, that programs notices that I’m not allowed to post, and generates an email warning me to that effect. These are some of the reasons SMTP is broken, but given the current state of SMTP, I don’t have any good ideas on how to prevent this type of back scatter.

In contrast, the messages today were bounced with mailbox full and mailbox unavailable messages. These type of messages should not generate back scatter. It’s 2008 and most mail servers are configured to reject messages for mailboxes that don’t exist or are over quota, relying on the sending server to generate the bounce back. Since spammers usually email directly to the receiving servers, they don’t bother generating the bounce back.

So why did I get these types of bounce backs? I carefully examined three messges and found one open relay (that’s an old problem) and two MX servers that accepted delivery to mailboxes and then turned around and generated an over quota message (in one case) and a generic delivery failed (in the other case).

The open relay problem is an age old problem - nothing new here. I suspect the other problems are due to people setting up much more elaborate mail systems, ironically, in an attemp to fight spam. The strategy is to setup a server (which will act as the offical MX server that is the first to receive the message) whose function is to just scan for spam and viruses and then pass the message on to the server that will deliver the email message. The problem is when that first line server does not know how to reject messages for mailboxes that don’t exist or are over quota. Instead, they accept all mail. If the server setup to deliver the messages can’t deliver the message - they have no way of communicating this rejection to the original sending server - so they either have to generate an email bounce, or the MX front line server has to generate an email bounce.

The final element I noticed was that the original emails being sent had forged Received lines.

Here’s one example:

	Received: from GMSEXC03.CORP.DS.GEMPLUS.COM ([10.2.158.51]) by
					GMSEXC05.CORP.DS.GEMPLUS.COM with Microsoft SMTPSVC(6.0.3790.1830);  Mon,
					24 Mar 2008 21:41:15 +0100
	Received: from SINEXC01.CORP.DS.GEMPLUS.COM ([10.150.204.51]) by
					GMSEXC03.CORP.DS.GEMPLUS.COM with Microsoft SMTPSVC(6.0.3790.1830);  Mon,
					24 Mar 2008 21:41:14 +0100
	Received: from mars.gemplus.com ([203.126.114.26]) by
					SINEXC01.CORP.DS.GEMPLUS.COM with Microsoft SMTPSVC(6.0.3790.1830);  Tue,
					25 Mar 2008 04:41:14 +0800
	Received: from 83-70-211-36.b-ras1.chf.cork.eircom.net
					(83-70-211-36.b-ras1.chf.cork.eircom.net [83.70.211.36]) by
					mars.gemplus.com (Postfix) with ESMTP id 1E3B62EACDF; Mon, 24 Mar 2008
					20:41:05 +0000 (GMT)
	Received: from [83.70.211.36] by leslie.mayfirst.org; Mon, 24 Mar 2008 20:41:05
					+0000

You’ll notice that the first received line (the one on the bottom) says that it was received by leslie.mayfirst.org from the IP address 83.70.211.36. The second recieved line says that it was received by 83-70-211-36.b-ras1.chf.cork.eircom.net also from the IP address 83.70.211.36. In other words, the first one, claiming to be recived by the mayfirst.org server, is clearly forged since the next line claims to have received the message from the same IP. All of this makes me very nervous about spam detection systems that block based on the original sending IP. Are they smart enough to see forgeries like this? This can be a very effective Denial of Service attack - scattering spam throughout the world claiming to have been originally received by the IP address of the mail system you want to target.

The Internet in 2008 without high speed access

Tue, 08 Jan 2008 09:32:44 +0000

Working on tech support in Guatemala City in October 2008 for the Social Forum of the Americas is having a profound impact on my assumptions.

The Internet and all my assumptions about tech are very different in a place without regular, high speed Internet access. It’s not possible to just say it’s like being in New York 10 years ago, because it’s not like that. The Internet I use regularly is an Internet based on an assumption that you have high speed Internet access - an assumption that didn’t exist 10 years ago.

Now, I’m re-thinking everything, like when and how often I install full system upgrades, the value of having installation disks that don’t need a network connection to install an operating system or program, and how our communications protocols should function if we assume inconsistent network connections rather than the opposite.

The state of linux on the desktop

Tue, 08 Jan 2008 09:32:44 +0000

In 2001 I was certain that the linux desktop revolution was just around the corner. It would hit in 2002, 2003 latest. Well, that didn’t happen. By 2005 I got tired of waiting and simply switched my desktop permanently to linux. At that point, I stopped paying close attention.

As Grace Lee Boggs pointed out during her closing key note at the 2008 Allied Media Conference, unlike uprisings or revolts, revolutions take time. They move slowly. So slowly, in fact, that we often don’t recognize or appreciate the amazing changes that have taken place.

I had that experience at Steven Mansour’s AMC presentation on using free and open source tools to make media. Steven demonstrated, with the audience participating, how to make flyers, images, audio programs, and even videos not only using free software, but running on an Ubuntu Linux laptop. In fact, Steven barely mentioned that he was running linux! There were occasional asides of “oh, yes, I think you can do that with a Windows or a Macintosh; not sure it would work as well though.”

Since I’m not up to date on multimedia applications, I learned a lot of practical information from the workshop. However, the most powerful realization that I took from the workshop was the amazing progress we’ve made in getting linux into the consciousness of the left.

The wonders of ps2pdf

Tue, 08 Jan 2008 09:32:44 +0000

I really don’t like printing. Most things I print just end up in the recycle bin or lost. However, there are times when it’s nice to have a physical copy of something (like making a card or a poster).

I just created a tabloid sized poster in the GIMP. I needed to convert it to a PDF so it could be printed. I first saved the file with a .ps extension, so the GIMP saved it as a postscript file. Then, I ran:

	ps2pdf file.ps

And voila, courtesy of the wonders of free software, file.pdf was created.

Unfortunately, the pdf displayed the poster in letter size (meaning a good portion was cut off).

Hm. How does this work? I fumbled around with my GIMP and evince settings (evince is my PDF viewer of choice) to no avail. Then I realized that I didn’t even know where the paper size is supposed to be set. In the pdf document itself? Or, does the PDF viewer need to be told? After some Internet searching, I discovered the pdfinfo command - which prints a summary of information about a PDF file. When I ran pdfinfo against my pdf file I got:

	Producer:       ESP Ghostscript 815.03
	CreationDate:   Sun May 25 12:46:12 2008
	ModDate:        Sun May 25 12:46:12 2008
	Tagged:         no
	Pages:          1
	Encrypted:      no
	Page size:      612 x 792 pts (letter)
	File size:      1108153 bytes
	Optimized:      no
	PDF version:    1.4

Ah - my pdf file itself is indicating letter size. I’m still not sure whether my original ps file contains any hints about paper site (although I read an interesting article on whether or not it should provide this information). However, I did find out that many programs use the libpaper library to determine paper sizes to use. With libpaper, your default paper size is configured in the file /etc/papersize. You can override this default by setting the PAPERSIZE environment variable.

ps2pdf seems to use libpaper, which is why it made my ps file into a letter-sized pdf file. I re-ran ps2pdf with a new value for the PAPERSIZE environmental variable:

	PAPERSIZE=11x17 ps2pdf file.ps

And sure enough, it worked:

	Producer:       ESP Ghostscript 815.03
	CreationDate:   Sun May 25 12:38:34 2008
	ModDate:        Sun May 25 12:38:34 2008
	Tagged:         no
	Pages:          1
	Encrypted:      no
	Page size:      792 x 1224 pts
	File size:      1108152 bytes
	Optimized:      no
	PDF version:    1.4

Ubuntu Gutsy (7.10) and bcm43xx wireless cards on Dell Vostro 1000

Tue, 08 Jan 2008 09:32:44 +0000

UPDATE: I spoke too soon! I’m leaving this full post for reference purposes, however, before trying to implement it - please seem comment below about how it didn’t work after all.

It has nearly killed me, but it finally seems to be working on the fix is not so difficult.

For starters - here are the details on the laptop I was using:

Dell Vostro 1000 Broadcom Wireless card (bcm94311MCG revision 1)

teacher@frick:~$ lspci | grep "Network controller"
05:00.0 Network controller: Broadcom Corporation BCM94311MCG wlan mini-PCI (rev 01)
teacher@frick:~$

There are two “things” that make the wireless card work. Often they are both referred to as “drivers.” One is the firmware. Firmware is software that is loaded into the wireless card by the kernel (at boot time). The other is the “driver.” The driver is the software that the kernel uses to communicate with the wireless card.

The firmware has to be copied to /lib/firmware. I don’t know any other way to load the firmware other than to reboot the machine, but there probably is a way to do it.

The driver is packaged as a kernel module and is loaded by the kernel.

Driver

The kernel I’m using (2.6.22-14) has drivers that work for me. They are called bcm43xx and you can see if they are properly loaded by running:

lsmod | grep bcm43xx

There’s a lot of directions out there saying that you shouldn’t use this driver and instead you should use ndiswrapper. I didn’t use those directions - the driver worked fine for me.

Firmware

Broadcom publishes the firmware, and it’s not free software. Therefore, one has to download the firmware from a web site that is authorized to distribute it and then run a program on your linux machine that can “cut” it into firmware files that your linux machine can use.

There are several versions of the firmware out there.

I started by using the Restricted drivers interface provided by Ubuntu, which installed bcm43xx-fwcutter and a shell script that downloads the firmware and “cuts” it into the right firmware files and puts them in /lib/firmware for you.

This didn’t work.

I then discovered a good site on linuxwireless.org which explained the difference between b43 and bcm43xx. I first moved the broadcom files that were in /lib/firmware to a backup directory. Then, I followed the directions on the linuxwireless.org, running the steps for b43, not bcm43xx (which is what the Ubuntu restricted drivers interface installed). Following the directions on that page are a pain in the butt. Wouldn’t it be nice if I could just post the extracted firmware files on this blog and you could just download them and put them in place instead of dealing with all that install this run that business? Too bad they are proprietary.

Anyway, it still didn’t work.

Then, on a random page about bcm43xx and Dapper I read:

VERY IMPORTANT: MOST PEOPLE NEED TO USE THE FOLLOWING COMMAND TO GET CONNECTED

sudo iwconfig ethX rate 11M

On my laptop, the wireless card was called eth0, so I ran:

sudo iwconfig eth0 rate 11M

Aha! Now it works. I’m still not sure if I needed the different firmware. Maybe the other firmware would have worked.

As the aforementioned page points out, you can add that iwconfig line to your /etc/network/interfaces file such that it looks somethin like this:

iface eth0 inet dhcp
wireless-rate 11M
wireless-essid My network
wireless-key 1234-5678-blah-blah-blah

URL vs URI

Tue, 08 Jan 2008 09:32:44 +0000

Every couple years I scratch my head and wonder what’s the difference between a URL and URI is. I inevitably google url vs uri and run through the first 5 hits or so and, after 15 minutes of reading explanation that don’t help, give up. I finally found an explanation that makes sense. Nothing like history to explain the present.

URLs That Make Iceweasel Crash

Tue, 08 Jan 2008 09:32:44 +0000

This is a work in progress. If you are running Iceweasel don’t click on these links unless you don’t mind a browser crash.

	Iceweasel version: 2.0.0.12-1
	URL: http://howtoforge.com/ipp_based_print_server_cups_p2
	Date: 2008-03-19

Using Git to collaborate

Tue, 08 Jan 2008 09:32:44 +0000

After spending weeks enraptured by the potential of git, I finally sat down and tried to setup a working environment in which two people could collaborate. It was harder than I thought.

The difficulty was compounded by a number of dumb mistakes in my setup. I finally figured it out with the help of Big Jimmy and by tailing my apache error log when trying to access my remote repositories to figure out what I was doing wrong (it showed a number of “file not found” messages).

Here’s a transcript of how I got it to work. I did all the work in a test environment on liberace (my laptop). I setup http://a.git-remote.liberace/ to point to a directory on my filesystem called a.git-remote and http://b.git-remote.liberace/ to point to a directory called b.git-remote.

# first create the initial collaborator's local repository:

0 jamie@liberace:git-test$ mkdir a.git-local
0 jamie@liberace:git-test$ cd a.git-local/
0 jamie@liberace:a.git-local$ git init
Initialized empty Git repository in .git/
0 jamie@liberace:a.git-local$ echo "a test" > test.txt
0 jamie@liberace:a.git-local$ git add test.txt
0 jamie@liberace:a.git-local$ git commit test.txt
Created initial commit 9461675: Test from a.
1 files changed, 1 insertions(+), 0 deletions(-)
create mode 100644 test.txt

# Next, create the initial collaborator's remote repository (this would
# normally be done via ssh). Notice how I move the initially created
# .git directory to a more reasonably named directory (is there a better
# way to do that with the git init command?).
#
# Also - notice how I chmod the post-update script - that causes git to
# update the server info every time a commit is made.

0 jamie@liberace:a.git-local$ cd ..
0 jamie@liberace:git-test$ git init
Initialized empty Git repository in .git/
0 jamie@liberace:git-test$ mv .git a.git-remote
0 jamie@liberace:git-test$ chmod 755 a.git-remote/hooks/post-update

# Now push local changes to the remote repo. This would normally be done
# over ssh
0 jamie@liberace:git-test$ cd a.git-local/
0 jamie@liberace:a.git-local$ git push ../a.git-remote/ master
Counting objects: 3, done.
adding b.
Writing objects: 100% (3/3), 223 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
To ../a.git-remote/
* [new branch]      master -> master

# now setup the second collaborator I use the git remote add
# command to add an alias so I don't have to retype the full
# URL every time. Maybe this step should be down with git clone?
0 jamie@liberace:a.git-local$ cd ..
0 jamie@liberace:git-test$ mkdir b.git-local
0 jamie@liberace:git-test$ cd b.git-local/
0 jamie@liberace:b.git-local$ git init
Initialized empty Git repository in .git/
0 jamie@liberace:b.git-local$ git remote add a http://a.git-remote.liberace/
0 jamie@liberace:b.git-local$ git pull a master
got 9461675b66f4e08cee6bee4e2f59736ff245ce64
walk 9461675b66f4e08cee6bee4e2f59736ff245ce64
got 970281b52e49f9bd0a70cf8dbf4fea49485d13ce
got 5fc6b4bc688dd10d9ad952716029ccaac0f2fb8e
0 jamie@liberace:b.git-local$ ls
test.txt

# second collaborator commits a change:
0 jamie@liberace:b.git-local$ echo "test b" >> test.txt
0 jamie@liberace:b.git-local$ git add test.txt
0 jamie@liberace:b.git-local$ git commit test.txt
Created commit 7f969da: adding b.
1 files changed, 1 insertions(+), 0 deletions(-)

# second collaborator creates remote repository
0 jamie@liberace:b.git-local$ cd ..
0 jamie@liberace:git-test$ git init
Initialized empty Git repository in .git/
0 jamie@liberace:git-test$ mv .git b.git-remote
0 jamie@liberace:git-test$ chmod 755 b.git-remote/hooks/post-update

# second collaborator pushes changes to remote repo
0 jamie@liberace:git-test$ cd b.git-local/
0 jamie@liberace:b.git-local$ git push ../b.git-remote/ master
Counting objects: 6, done.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (6/6), 445 bytes, done.
Total 6 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (6/6), done.
To ../b.git-remote/
* [new branch]      master -> master

# initial collaborator pulls in second collaborator's changes
0 jamie@liberace:b.git-local$ cd ..
0 jamie@liberace:git-test$ cd a.git-local/
0 jamie@liberace:a.git-local$ git remote add b http://b.git-remote.liberace/
0 jamie@liberace:a.git-local$ git pull b master
got 7f969da23ffb57b70405a9e9fc62ae39cedd0f1b
walk 7f969da23ffb57b70405a9e9fc62ae39cedd0f1b
got 52f8ddad8dc10896059f68a48d4cb3b3175556ed
got 8f9ad5050475fa0981c9d0f911381ffa7ddb7c51
Updating 9461675..7f969da
Fast forward
test.txt |    1 +
1 files changed, 1 insertions(+), 0 deletions(-)

# lo and behold! the chages are present:
0 jamie@liberace:a.git-local$ cat test.txt
a test
test b

Web 2 point... well you know

Tue, 08 Jan 2008 09:32:44 +0000

We’re at the Allied Media Conference - day 1! Stay tuned for more blogs about what I’m seeing.

This is the first session on day one present by Geoff Hing. Despite the name of the session (ug - jargon hell!) - it was an interesting workshop. A few thoughts:

Tags as non-hierarchical approach to information organizing. I’ve never really considered the political implications of free tagging versus hierarchical categorizing. It really pushes the power of creating meaning toward the users rather than the administrators. In the worst scenarios it means everyone is an independent agent defining their own reality - the worst aspect of liberalism. On the other hand - that’s where organizing comes in - to organize our own meaning (in this context that means collaboratively defining tags).
Use of device independent communication. Geoff’s response to to the often stated problem “but not everyone has access to the Internet” is: then use other ways of communicating, such as cell phones (twitter) or even landlines (jot) to bridge the gap.
Use of API (application programmer interface) as means of decentralization.

It was a real relief to go to a Web 2.0 presentation that analyzed the concepts from a political perspective.

The political discussion and conclusions, however, are the same ones we’ve been going over and over: how do we use the Internet when not everyone has access to it or when it’s too difficult for some people. I don’t want to detract from this problem - it’s a real one. On the other hand - I think we tend to address it too blindly in ways that lead to bad political decisions, such as using corporate services because they’re “easy” and “more accessible.” I think political work is struggle - and yes, we do need to struggle with this technology.

Web stats and robots

Tue, 08 Jan 2008 09:32:44 +0000

I was analyzing the web traffic of some of our most popular member sites to try to understand how to better handle high traffic and I found an interesting statistic. Here are the top four IP addresses in the web logs of one member’s web site (over a period of five days).

	IP ADDRESS        NUMBER OF LINES IN ACCESS LOG

	66.249.67.154     61729 
	74.6.8.113        47576 
	74.6.8.107        44039 
	83.231.136.9       6560

What we’re seeing are three IP addresses that are dominating the site with between 7 and 10 times the number of hits than any other IP address. Who are these people?!? Well, the first one identifies itself as Google and the second two as Yahoo. In other words, they are robots feeding search engines. The fourth looks like a regular web browser (running Opera!!).

Welcome to my new blogging software

Tue, 08 Jan 2008 09:32:44 +0000

Times are a-changing and so is my blog software. May First/People Link is about to launch its new blog site, which means we’ll no longer being blogging on the main www.mayfirst.org site.

I will be re-publishing my blog on the about to be launched May First/People Link blogging site. However, I decided to take advantage of this opportunity to play around with ikiwiki. It’s particularly interesting because it stores my blogs in git - a revision control system often used for software projects. That means I get to write my blogs in a normal text editor rather than a web form.

I did my best to bring over my old blogs - including comments! The order is a bit off - if you look at the [[posts]] page they are not listed in chronological order. I decided I could live with it. I hope you can too!

Welcome to the Monkeysphere!

Tue, 08 Jan 2008 09:32:44 +0000

After months of coding and testing, the Monkeysphere had it’s first public announcement.

Yippee!

You can read the announcement for the technical details (or just go to the web site itself).

For me, the most important aspect of the Monkeysphere is: placing security in the hands of us, the users. The Monkeysphere is another extension of the web of trust, perhaps the most powerful approach to security in a densely populated world. With the web of trust we build our own networks of trust: I assign trust to my friend Jose. If Jose verifies someones identity, then I trust that verification. And so on.

The deep dark secret on the Internet, particularly for folks on the left, is that most Internet security systems operate differently. Using an alternative system called x509, they rely on the entire Internet designating a limited number of “certificate authorities” to verify the identities of the people we work with. Most of these authorities are for profit corporations. But even if they weren’t for profit, why a hierarchical model? Is this the world we want to build on the Internet?

The Monkeysphere introduces the ability to use the web of trust when we connect to servers using secure shell or secure FTP. While the use secure shell and secure FTP is mostly limited to system administrators and web developers, the introduction of the web of trust into this aspect of the Internet is a major and exciting first step toward introducing it even further, into security contexts used by everyone.

What is acceptable security?

Tue, 08 Jan 2008 09:32:44 +0000

The security we use to protect May First/People Link members is not perfect, but we try pretty hard with what I would consider to be a good results.

I’m constantly amazed to see how much better capitalized industries and companies fail to take even basic steps to secure their systems.

Consider the banking industry. We’ve always known that the information that is on our credit cards - the same credit cards we hand over to countless people, the same information we provide over the phone and the Internet to yet more people, is all the information anyone needs to take our money. This should not come as a surprise, given our history with checks. The information on a single check is enough to compromise ourselves financially (thanks dkg for pointing out an interesting articles on the topic of identity theft via routing numbers.

The other day I read an [article about a credit card breach]((http://www.nytimes.com/2008/03/23/us/23credit.html) due to information be stolen in transit between computers. My favorite quote was: “Wider use of encryption might seem an obvious answer. But in practice, encryption is unused at certain points in a data-processing chain because the computing power it requires can slow transactions.” Hm. I guess it depends on where you priorities are.

This morning I had yet another experience - this one personal. I realized that I didn’t have the flight number or confirmation of a flight I bought for my brother. I called the American Airlines phone number and, talking to their automated voice thingy, I gave the city and time of the departure and my brother’s last name and AA confirmed his flight. I had my credit card ready so I could enter the last four digits to prove that I was the one who bought the ticket - but apparently that’s not necessary. I considered calling back and saying a number of different last names to see if I could build a list of passengers.

What's wrong with the service economy

Tue, 08 Jan 2008 09:32:44 +0000

When trying to explain May First/People Link to people I often revert to saying something along the lines of: “We’re kinda like an Internet Service Provider.”

From now on, I’ve decided never to use the word “service” again when describing us.

In fact, I’d like us to work on a statement that all people who join May First/People Link must agree to that says: “May First/People Link does not engage in service relationships.”

I would even extend this statement beyond May First/People Link to say: We should never engage in service relationships when doing political work on the Internet.

What do I mean by service relationship? Here are a few common characteristics:

Tunnel visioned - aside from pleasantries, a service relationship is focused exclusively on the task at hand. There’s no room for spontaneity, creativity or taking advantage of everyone’s multi-faceted skills and experiences.
Competitive - if not explicit, there’s always an under current of competition between the client and the provider around what the client can take and what the provider will give. It often focuses on who controls what is given, gaming the system and pulling rank, leaving no room for collaborative activity.
Individuated - the client is given the impression that (and is expected to act as if) they are the only client in the world. There is nobody else and nothing more important than the client. In short, with a service relationship there is little if no room for solidarity, mutual aid, creativity, transformation or any other qualities essential for building social and political change in the world.

In a perfect world, I can’t imagine why we would want any service relationships. What does that say about the overwhelming trend in the United States toward a service economy? How can we resist that trend?

Given the existing world, it’s not surprising that we have a lot of service relationships. We can’t change everything at once. Within the left, our least complicated and least important relationships are service relationships and will probably remain that way for some time to come: electricity, accounting, rent, post, etc.

But wait - electricity delivery, for example, is both complicated an important. How can I put that in the least important and complicated category?

When I say “complicated” and “important” - there’s distinction between what is being provided and the relationship between the people and organizations involved. In other words, it doesn’t matter how great you get on with your meter reader - it won’t have much impact on your organization or mission.

Your relationship with your technology and Internet partners on the other hand will have a huge impact on how you advance your goals and objectives.

Let’s face it, those of us on the left who think that technology and the Internet “should just work” are living in a world of denial. Organizing on the left is increasingly about how the Internet works for our particular (and changing) objectives. And, given the complexities involved, we find ourselves increasingly reliant on people with particular skills on the Internet to help us work that out.

In this situation, the kind and quality of that relationship is critical. Is it a service relationship (with all the limitations described above) or is their a different relationship model it should be built on? Is it a long term relationship, worthy of spending time building a foundation, or is it a short term relationship with limited objectives? Is it a relationship of getting, or a relationship of mutual aid and collaboration?

This list of questions could go on and on. And, they are as important for people seeking technology and Internet partners as they are for those of us able to share that experience, since all parties participate in determining what kind of relationship we are having.

Building a new type of relationship around organizing on the Internet will take a lot of work. Here are a few ideas that I think are important if we are to be successful:

Decisions on who to partner with should be based on political compatibility, not money, personality, or who is available now.

By “political compatibility” I don’t mean running some kind of simple questionnaire or political litmus test. I work with a lot of people who don’t share my politics. However, a successful relationship should start with mutual respect for the goals of each partner and the goals of the project. If the project is to build a system to drive a get out the vote campaign, then the technology partner must value and support that goal. If the technology partner has an ideological commitment to free software, than all partners should respect and value those beliefs.

The key to achieving this goals is to develop a community of technologists who can make a reasonable and, most importantly, stable living while partnering on these projects. I make no assumptions here: it doesn’t mean technologists have to get paid directly for working for the left or we need to create some kind of network of small businesses. Maybe that would do it, maybe not. Maybe we need to advance our technology skills dramatically - so that at any given political meeting, we can pull together a respectable technology team in which no one or two people will get saddled with everything. Who knows. But we need to figure something out because at the moment building technology partnerships are often acts of desperation due scarcity of technologists who can work at rates that the left is paying.
Recognize what’s going on: technology and the Internet is not a product.

Despite mainstream conceptualization of technology and Internet relationships as service relationships, we also somehow think of them as involving a product. We release Requests for Proposals (RFPs) which describe what we want and expect bids that will say how much it will cost to deliver what is requested in the RFP. On the technologists end, we think we’re going to “wrap up” the project next week, freeing us up to deliver the next project. It’s remarkable. Even those of us who have done this work for years and years often insist in our heads (or out loud) that we’re either going to “get” that web site or database in two weeks or that we’re going to “hand over” the project on the deadline. We know that never happens. No project is every done (unless the relationship ends). The beginning of every project is the beginning of a relationship.

Money has something to do with it. Often volunteer projects are much more relationship focused that product focused. Similarly, projects in which I’m paid a monthly stipend (as opposed to a flat fee) have a very different nature to them. However, there are plenty of exceptions.
Mutual support

All partners must support each other’s projects. That might be a technologists offering non-technical strategic ideas about the campaign. It might be an organizer pointing out technical flaws of the project in the context of helping the growth and development of the technologists. In either case, the relationship has to be supported as much as the project itself.

YouTube for the left?

Tue, 08 Jan 2008 09:32:44 +0000

The Miro project has come a long way from the Democracy Player I remember from years back. The folks at the Participatory Culture Foundation have re-organized the project from a media player to something many times more powerful: they are building a radically decentralized and democratic system for sharing video content.

They demonstrated Miro at the 2008 Allied Media Conference. Essentially, it is a program that organizes the world’s video feeds and helps us all find and watch them.

Wow.

So what exactly does that mean?

When you download Miro, by default it comes installed to use the collection of video blogs that have been submitted to the Miro Guide. Anyone can submit a video feed to the guide and pending a review (can’t seem to find any documentation on how the feeds are reviewed) it will show up in the guide.

However, there’s no reason you need to use the Miro Guide. You can specify any other site to pull your content from and you can individually add your own feeds. In fact, the Miro folks have gone out of their way to de-brand the program. They even offer a way to brand your own miro player.

Let’s think about why this is 100 times better than YouTube:

Decentralized. There is no one web site where everyone has to upload their video that can be taken down, sold, crash or can go out of business.

Licensing. You are not handing over video content to anyone except the server storing your video - and you can choose to store your video where ever you want. You only need to publish the feed on Miro Guide. Ever read the YouTube terms of service? My favorite part:

 ... by submitting User Submissions to YouTube, you hereby grant YouTube a
 worldwide, non-exclusive, royalty-free, sublicenseable and transferable
 license to use, reproduce, distribute, prepare derivative works of,
 display, and perform the User Submissions ...

Standard protocols. Miro runs on RSS - already a bedrock, standard protocol.

Politics. The project is engineered to prevent the Participatory Culture Foundation from perverting its democratic potential. The software is not only free/open source, but it is designed to give equal footing to any and all content providers. That, I think, is the best indication of any groups politics. In addition, they’ve structured their organization in a way that keeps them honest:

 People often ask why we're setup as a non-profit rather than a for-profit.
 Quite simply: all of us at PCF are drawn to the project because of the
 mission and being a non-profit is the only way we can ensure that the
 mission is built into the structure of the company. So many times we've
 seen for-profit companies lose their values as financial pressures mount,
 founders leave, or they get acquired. We want to make sure that can't
 happen.

 Being non-profit has other benefits as well. Most importantly, it means
 that we are accountable to our user community and the public. There aren't
 any venture capitalists or shareholders that can force us to go in a
 direction that's bad for users but good for profits.

And lastly, they declare their mission which explicitly states their commitment to openness.

On the last point … from a radical movement perspective, I don’t want to over estimate their politics - this is a decidedly liberal organization firmly rooted in, and limited by, the foundation dominated non-profit world. However, the core values that form the basis of the project are core values that I share and provide a powerful basis for collaboration with the left.

Miro is a project we should all be behind 100%!

And now munson gets Etch

Mon, 08 Jan 2007 09:32:44 +0000

Learning from my peewee experience…

After editing /etc/apt/sources.list I started with:


mayfirst@munson:~$ sudo apt-get install initrd-tools

Which gave me:

mayfirst@munson:~$ sudo apt-get install initrd-tools
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
libc6 libc6-dev libdevmapper1.02 libselinux1 libsepol1 locales tzdata
Suggested packages:
glibc-doc
The following packages will be REMOVED:
base-config
The following NEW packages will be installed:
libdevmapper1.02 libselinux1 libsepol1 tzdata
The following packages will be upgraded:
initrd-tools libc6 libc6-dev locales
4 upgraded, 4 newly installed, 1 to remove and 194 not upgraded.
Need to get 12.0MB of archives.
After unpacking 1810kB of additional disk space will be used.
Do you want to continue? [Y/n]

So far so good..

Then I tried:


apt-get install --purge linux-image-2.6-686

The purge part is because I want to purge hotplug. Which produced:

mayfirst@munson:~$ sudo apt-get install linux-image-2.6-686
Password:
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
busybox initramfs-tools klibc-utils libklibc libvolume-id0
linux-image-2.6.18-4-686 lsb-base makedev module-init-tools udev
Suggested packages:
linux-doc-2.6.18
Recommended packages:
libc6-i686
The following packages will be REMOVED:
hotplug*
The following NEW packages will be installed:
busybox initramfs-tools klibc-utils libklibc libvolume-id0
linux-image-2.6-686 linux-image-2.6.18-4-686 lsb-base udev
The following packages will be upgraded:
makedev module-init-tools
2 upgraded, 9 newly installed, 1 to remove and 191 not upgraded.
Need to get 17.3MB of archives.
After unpacking 52.0MB of additional disk space will be used.
Do you want to continue? [Y/n]

Next comes the reboot

And then:


apt-get dist-upgrade

And away we go.

Apache and ldap authentication

Mon, 08 Jan 2007 09:32:44 +0000

The project at hand: Setup an LDAP directory that will work with Apache2 using basic auth. In other words, I want to protect Apache directories by forcing users to enter a username/password (using http basic auth) and have the username and password checked against an ldap server.

This has taken me a week and many many hours, but is finally up and running.

I’m running Debian Etch and started with:

apt-get install slapd

I took all default options and, when prompted, entered:

office.example.org

As my domain

And:

Example Organization

As the name of the organization.

I also entered the password for the admin user.

Without doing anything else, I then tested the setup by running:

ldapsearch -x

The -x means use simple authentication. If you don’t specify -x you’ll be prompted for a SASL password. If you do specify it, you won’t be prompted for a password at all - instead you’ll be logged in automatically as an anonymous user.

On debian, by default, the anonymous user has read access.

My first problem was that after doing this very basic step, I got the following error:

No such object(32)

I spent hours googling around before I found the problem: I didn’t specify the “base dn” to search.

The solution to this problem is to either add the base search dn to your ldapsearch command with -b “cn=admin,dc=office,dc=example,dc=org”

ldapsearch -x -b “cn=admin,dc=office,dc=example,dc=org”

or, better yet, to add the following to /etc/ldap/ldap.conf:

HOST 127.0.0.1 BASE dc=office,dc=example,dc=org

That file is read every time you run ldap.

Next, I wanted to add some information to ldap. This step required some new conceptual understanding that I was lacking. Coming from a sql background, I had to do some translation just to understand what was going on.

One way of thinking about ldap is that you have to first create your container (loosely like creating tables) before you can create objects in those containers. I chose to create an “organizational unit” called People - so that I could add all my users in this organizational unit, allowing me to create other units (like computers, etc.) if I wanted to.

First I created that organizational unit by creating a file called: create-people-ou.ldif with the following contents:

# Setting up container for Users OU
dn: ou=People,dc=office,dc=example,dc=org
objectclass: top
objectclass: organizationalUnit
ou: People

I added this definition to the ldap server with the command:

ldapadd -x -W -D “cn=admin,dc=office,dc=example,dc=org” < create-people-ou.ldif

The -x, as you recall, means use simple authenticatin. The -W means prompt for a password. The -D means bind using the following dn. The argument following is the dn that represents the admin users that the debian install script gave full write access to. Then, you redirect the file you created into standard in and, after being prompted for and entering the password you created via the debian installation, you should be in business.

Check with:

ldapsearch -x

Then, I populated my ldap server with a single record. I created a file called add-user.ldif with the following content:

dn: cn=Jamie McClelland,ou=People,dc=office,dc=example,dc=org
cn: Jamie McClelland
objectClass: shadowAccount
objectClass: inetOrgPerson
givenName: Jamie
sn: McClelland
uid: jmcclelland
mail: jm@not.mayfirst.org
userPassword: {crypt}jfjkle234

The string after {crypt} is the crypted version of the password I want. I’m still trying to figure out how to get md5 to work.

I added this record with:

ldapadd -x -W -D “cn=admin,dc=office,dc=example,dc=org” < add-user.ldif


Again, you can test with:
ldapsearch -x
And you should see this new entry.
If you want to test the password creation (and demonstrate that this new user indeed can authenticate) try:
ldapsearch -x -W -D “cn=Jamie McClelland,dc=office,dc=example,dc=org”
When prompted enter the password you created in your add-user.ldif file.
If it works, you’re in business.
Next, tried to get apache2 to work with ldap. Ldap auth comes with apache2-common, so no new modules needed (another hour wasted to learn this).  But you do have to enable it with:
sudo a2enmod authnz_ldap
Next, I added the following to my vhost definition:
AuthzLDAPAuthoritative off
AuthBasicProvider ldap
AuthType Basic
AuthName "Test Jamie"
AuthLDAPURL ldap://localhost/ou=People,dc=office,dc=harlemrbi,dc=org
require valid-user

Well, I didn’t add that at first. I went through the following:
[Mon May 07 13:12:48 2007] [error] Internal error: pcfg_openfile() called with N ULL filename
[Mon May 07 13:12:48 2007] [error] [client 192.168.0.150] (9)Bad file descriptor : Could not open password file: (null)
This error will pop up if you leave off:
AuthBasicProvider ldap

With apache 2.2 you have to specify that, or apache will try to use the default file-based basic auth.
Then, I entered the right password via the http basic auth login, no errors in the apache error log, but, rather than getting into my directory, I was presented with the login again. It seemed as though ldap was properly authenticating, however, apache was still not letting me in.
That was because I hadn’t included the line:
AuthzLDAPAuthoritative off

The “require valid-user” uses another auth module. So, I have to say: don’t make ldap authoritative - allow other modules to pitch into this authentication effort.
And finally it is working.
One more note. When troubling shooting ldap, you may want to change the loglevel in your /etc/ldap/slapd.conf file from:
loglevel        0

to
loglevel        any

Then, check syslog for a gazillion log messages.



AWAAM Responds
Mon, 08 Jan 2007 09:32:44 +0000
MFPL member Arab Women Active in Arts and Media are now speaking out to address reports about a t-shirt they produced bearing the slogan, “Intifada NYC,” displayed at an Arab heritage festival last month. As you can imagine, the media has done a typical hack job on the story - please check out their version on their website.


Backupninja running out of disk space
Mon, 08 Jan 2007 09:32:44 +0000
Backupninja starting giving me errors along the lines of:
[Errno 28] No space left on device Fatal Error: Lost connection \
to the remote system

Yet, df on that server showed plenty of room:
0 munson:/var# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda1              1829159    663080   1068486  39% /
tmpfs                    95492         0     95492   0% /lib/init/rw
udev                     10240        48     10192   1% /dev
tmpfs                    95492         0     95492   0% /dev/shm
/dev/mapper/main-var    982728    216664    713636  24% /var
/dev/mapper/main-srv 206424760  69507240 126431760  36% /srv
0 munson:/var

All backups were going to the /srv partition, which has a lot of room, both before and after the backup started and ended.
Next I ran:

watch df

To see how it changed in realtime while the backup was running, and I found that it’s the var partition that was growing. Fortunately, backupninja was cleaning up after itself when it quit, leaving the filesystem exactly the same as when it started.
I wanted to know what directory was getting written to, so I tried running:

find . /var > /root/before.txt

Before running the backup, followed by:

find . /var > /root/after.txt

During the backup.
However:

diff -u /root/after.txt /root/before.txt

Revealed no changes.
So, I tried:

lsof | grep rdiff

And found, among other things:
rdiff-bac  7624 rbi-backup    7u      REG      254,2  231211008   \
14172176 /srv/tmp/tmphgYI7a (deleted)

It looks like rdiff backup was creating a temp file using the /var/tmp directory.
Since this server only serves one purpose: backing up, I decided to make things easier and simply move the /var/tmp directory to /srv/tmp and then leave a symlink in /var/tmp.
Problem solved.


Coalition of Immokalee Workers: major mobilization in Chicago April 13 - 14
Mon, 08 Jan 2007 09:32:44 +0000
    
May First/People Link member [CIW](http://ciw-online.org) needs your help publicizing their major moblization in Chicago on April 13 and 14! I've reprinted their announcement below. Please circulate it widely and consider supporting them. CIW has been doing amazing work around food/agriculture, including work around the treatment of farmworkers by Taco Bell and McDonalds.

The Coalition of Immokalee Workers announces a major mobilization for
farmworker justice, April 13-14, 2007, in the greater Chicago area.
Make your plans to join us for historic actions that will usher in a
new phase in the Campaign for Fair Food.
CIW: “Today, we are tired, in the words of Martin Luther King Jr., of
`relying on the good will and understanding of those who profit by
exploiting us.'”
Nearly two years have passed since Taco Bell and the CIW announced an
historic initiative to address the ever-deepening poverty and decades
of degradation faced by farmworkers in Florida. At that time, Taco
Bell challenged its fast-food industry counterparts to join in
demanding fair wages and humane treatment for the workers who pick
their tomatoes.
McDonald’s, the undisputed leader of the $100 billion fast-food
industry, has refused to meet that challenge. Despite increasing
public pressure on the fast-food giant, McDonald’s has refused to
recognize the seriousness of the exploitation of tomato pickers
exposed through the Taco Bell boycott, and refused to work with the
CIW to address that exploitation. Instead, it has taken measures that
appear aimed at undermining the hard-won advances in wages and working
conditions established in the agreement with Taco Bell.
Today, after nearly two years of waiting patiently for McDonald’s to
join us in addressing the crisis of human rights abuses and
sub-poverty wages in its tomato supply chain, we are tired of waiting.
We are tired, as Martin Luther King Jr. said, of “relying on the good
will and understanding of those who profit by exploiting us.”
Our members and allies feel that it is now time to intensify our
efforts. As such, the campaign is entering a new phase this year, and
support from our allies across the country will be more important than
ever. The action in Chicago this April will mark the first major
engagement in this new phase of the Campaign for Fair Food.
Mark your calendars today and start organizing to bring members of
your community to McDonald’s backyard this April to join us in a:


Major rally outside McDonald’s global headquarters in Oak Brook, IL,
Friday, April 13, 2007.


Carnaval and Parade for Fair Food, Real Rights, and Dignity -
Saturday, April 14, 2007 in downtown Chicago.


We strongly encourage our allies from across the country to join us in
Chicago this April and to participate in the colorful carnaval and
parade action. Groups are encouraged to organize their own floats and
delegations for this action. Contact workers@ciw-online.org for more
info.
For background, campaign and action updates, and more information,
visit http://www.ciw-online.org .
See you in Chicago! Thanks, Coalition of Immokalee Workers


Deep thoughts about timestamps and time zones
Mon, 08 Jan 2007 09:32:44 +0000
I’ve been struggling with Drupal’s timezone/daylight savings handling (see Drupal discussion).
In the course of trying to get things to work right, I gave the concept of timestamps a much closer look.
For starters, it really dawned on me that a timestamp, by definition, is NOT the number of seconds since January 1, 1970. It’s the number of seconds since January 1, 1970 UTC (which is also known as GMT).
In other words, if I were to create a timestamp here in New York that represents January 1, 1970 at 00:00:01 EST (one second after midnight), the time stamp would not be “1” as in one second - it would be: 18001. In other words, 60 seconds times 60 minutes * 5 hours (to account for the fact that today I am 5 hours behind UTC) + 1 second:
60 * 60 * 5 + 1 = 18001

PHP handles all of this conversion for us, which is part of the reason I’ve been so blissfully unaware of how complicated things are.
When I use php to create a timestamp, it creates a timestamp representing my date/time in UTC. When I use php to convert a timestamp to a more human readable form, it converts it back to my timezone:
0 jamie@liberace:~$ php -r 'echo mktime(0,0,1,1,1,1970) . "\n";';
18001
0 jamie@liberace:~$

0 jamie@liberace:~$ php -r 'echo date("Y-m-d H:i:s",18001) . "\n";';
1970-01-01 00:00:01
0 jamie@liberace:~$

If you want to get the actual UTC timestamp for a date, use the gmmktime and gmdate functions instead.
Also of note - the world’s easiest way to convert a timestamp to a local, human-readable date:
date -d @18001



Dell 1720dn printer PPD file
Mon, 08 Jan 2007 09:32:44 +0000
What a struggle. All I wanted was a working PPD file for the Dell 1720dn. I searched google high and low without luck. Finally I found a reference to a Windows PPD file for the 1710. It was on driverguide.com, which made me jump through flaming hoops to download it. It’s attached to this post and seems to work ok for the 1720dn.


Drupal and phpmailer
Mon, 08 Jan 2007 09:32:44 +0000
Thanks MFPL member Jesse Mortenson for finding a great set of simple instructions that explain how to get Drupal to use phpmailer (specifically so that Drupal will send email via an alternate smtp server, rather than localhost).
Details here.


Drupal Anonymous user missing
Mon, 08 Jan 2007 09:32:44 +0000
For the second time in two weeks we’ve had a Drupal problem related to user access and the anonymous user. Both sites are running Drupal 4.6.11. It goes something like this:
Suddenly we have errors like this one:
	Fatal error: Duplicate entry '6e00c2d43650a9a6a00a23a73d5dee11'
	for key 1 query: INSERT INTO sessions (sid, uid, hostname, timestamp)
	VALUES ('6e00c2d43650a9a6a00a23a73d5dee11', 0, '69.131.184.249',
	1180063799) in
	/usr/local/share/drupal-4.6.11/includes/database.mysql.inc on line 66

Or - regular pages (like the home page) suddenly display access denied messages like:
	Access denied
	You are not authorized to access this page.

Upon looking in the database we discover that there is no longer a user with UID 0. That’s the uid reserved for the anonymous user and Drupal sites will crash and burn in all kinds of weird ways without it.
Equally distrubing we discover that there is a new user (recently inserted according to the time stamp field) with a different UID, but without a name or email value. The anon user typically has no name and the name field is unique: you can’t have two users with no name.
So, we get rid of the weird user and add the anonymous user with uid 0 to the users table. Then we’re faced with an Access Denied message.
How the Anon user was removed and the user without a name or address was inserted is my biggest concern to prevent this from happening again. However, my immediate task is to get the site operational.
So… these are the steps I took to debug the Access Denied message.
We have a central drupal installation that all of our members use. In order to trace where the access denied message is coming from, I have to tweak the central code. I don’t want to do that for all members, so my first step was to backup this site’s web directory (which contains symlinks to our central Drupal installation) and then re-create their web directory with a copy of the Drupal code. This way I can tweak the drupal code without affecting other members.
Next, I added a function to the settings.php file:
	function mf_output($output,$die = false)
	{
	  if($_SERVER['REMOTE_ADDR'] == '1.2.3.4') print_r($output);
	  if($die) die();
	}

This function will let me output debugging info, but just for me (or anybody else at my IP address). Replace 1.2.3.4 with your IP if you want to do this.
Then, I grepped the code for “access denied”:
	grep -ir "access denied" *

I found a solid reference in include/common.php. There’s a function called drupal_access_denied that seems to control that feature.
Next I looked for references to that function:
	grep -ir "drupal_access_denied" *

Lots of modules use this function. I first checked out modules/node.module.  Before each call of that function I added:
	mf_output(1);

I then reloaded the browser - no luck. The message wasn’t coming from the node module.
Then, I discovered that index.php calls it. I used the same technique and discovered that the function menu_execute_active_handler is returning “MENU_ACCESS_DENIED”. Wonder why that is happening. The function menu_execute_action_handler is in the includes/menu.inc file.
The relevant code is:
	if (!_menu_item_is_accessible(menu_get_active_item())) {
	  return MENU_ACCESS_DENIED;
	}

So I continued looking for the _menu_item_is_accessible function.
The first thing it does is build a menu array with:
	$menu = menu_get_menu();

Dang. This is a complex function that builds the menu from all the modules.
I started by inserting this in the function:
	mf_output($menu,true);

That produced an insanely huge array. The _menu_item_is_accessible function seems to look for a key in this array called ‘access.’ The insanely huge array has the key set to either 1 or it’s unset (in most cases). Where is that supposed to be set?
At this point, I decided to check out the user.module, which is the module that defines what users have access to what permissions. In the course of looking at the user_access function, I noticed it pulling from the users_roles table. When I looked more closely at this table, I realized that there was no entry linking the anonymous user (uid 0) to the anonymous role (rid 1). So I added a record to the users_roles table with uid 0 and rid 1. Then, I emptied the cache table and now it seems to be working again.
This seems to support the theory that the anonymous user was in fact deleted by drupal code - because the users_roles entry was elegantly removed as well, as opposed to the new weird user be created via an update of the existing anonymous user.


Drupal Upgrade Disaster
Mon, 08 Jan 2007 09:32:44 +0000
We upgrade a Drupal site from 4.6 to 4.7 and to my pleasant surprise everything went smoothly… until 2 weeks later the administrator of the site noticed that several blog entries were truncated at seemingly random places. He sent me 10 examples of blogs that were anywhere from 10 to 90% of their original length, all of them cut-off in the middle of sentences. Furthermore, the content was missing in the MySQL database - it was not just a problem with php or the browser not rendering the content properly.
After carefully looking at the entries I noticed that all of them had “smart quotes” in them. When I compared the truncated blog entries with the original entries (from the 4.6 backup) I noticed that they were all cut-off just where the closing smart quote should be.
Next, I tried to search the 4.6 backup for all nodes that have smart quotes. This was no easy feat because the smart quotes are represented with weird binary characters. After some struggle and googling, I discovered that using the hex function was the easiest way to do it.
One of the affected entries had
emergencies.[binary character]? The
Then I did:
SELECT HEX(’emergenices.’);
SELECT HEX(’? The’);
And finally:
SELECT HEX(body) FROM node WHERE nid=350;
I then searched for the output of the first two select statements and discovered that the hex version of the closing smart quote is:
E280
Then:
SELECT nid FROM node WHERE body regexp UNHEX(‘E280’)
And that returned all the articles, including the ones that were not truncated.
More examination by hand revealed that the problem only happend with closing smart quotes followed by a question mark. The hex value for a question mark is:
3F
Howevever:
SELECT nid FROM node WHERE body regexp UNHEX(‘E2803F’)
Doesn’t work because the question mark is a special regular expression character. Finally I got this to work:
SELECT nid FROM node WHERE body regexp concat(unhex(‘E280’),’\?’);
About 10% of all the nodes were affected.
Next I created a table to only store the affected data:
CREATE TABLE node_original ( nid int(10) unsigned NOT NULL, body longtext NOT NULL);
And then I selected the affect nodes into that table:
INSERT INTO node_original (nid,body) SELECT nid,body FROM node WHERE body REGEXP CONCAT(unhex(‘E280’),’\?’);
Next step: I wrote a php script designed to be written from the command line. It bootstraps into the Drupal installation, pulls the data from the node_original table and updates the node in the drupal site (making a revision in case something bad happens).
It’s attached to this blog in the event that someone might find it useful. NOTE: please don’t run it without fully reading it! It could do very bad things.


Drupal, getid3 and Debian
Mon, 08 Jan 2007 09:32:44 +0000
GetID3 is a php library that reads MP3 files and extract information about the artist, song, etc. It’s used by the Audio file when uploading an image to automatically fill in the artist name and song information.
Last month, due to a Drupal Security Advisory we removed the getid3 library directory from all the audio modules installed on our systems. The maintainer of the audio module included the getid3 library in the audio module itself. In addition the maintainer included the entire tar ball of the getid3 library, including sample scripts. Drupal released an advisory because someone figure out how to manipulate drupal into doing things with the included sample scripts. So - my sledge hammer approach was to remove all the getid3 library folders from all the audio modules on our system.
Now fortunately, the audio module maintainer included a drupal variable where you can say the path to your own getid3 library. It’s in:
Admin -> Settings -> Audio -> GetID3
So, for Chavez, we installed the Debian package for php-getid3. If you’re primary host is chavez.mayfirst.org, then you should set this directory as:
/usr/share/php-getid3
Viewsic, however, does not have php-getid3 (since viewsic is running the
sarge version of Debian and Chavez is running the etch version). So if you’re primary host is viewsic.mayfirst.org, you should use:
/usr/local/share/php-getid3


Facebook's idea of privacy
Mon, 08 Jan 2007 09:32:44 +0000
Facebook retreated from it’s Beacon program thanks to the campaign organized by MoveOn.
The Beacon program automatically publishes on your Facebook page (and the Facebook pages of your friends), and as an RSS feed, purchases that you make with participating web sites like Fandango or Travelocity or Amazon.
In other words, the moment you finalize that Amazon purchase of “Want a New, Better, Fantastic Job?: How to Find Satisfying Work in This Tipsy-Turvy World” off it goes to the world (inluding your boss). The hubbub is about whether and how easy it is to opt out of this progrm.
I fully understand MoveOn’s strategy of targeting Facebook - they are the single entity publishing this information. They’re doing a bad thing and their an easy target. However, it makes me wonder about every single third party web site that has agreed to this program. Facebook couldn’t do this if Amazon said: “No thanks. We won’t provide you with that information. It’s private and belongs to our customers.”
What are the privacy policies for Amazon that allow them to report to Facebook everything that I buy?


Failure to mount root partition after major lenny dist-upgrade
Mon, 08 Jan 2007 09:32:44 +0000
I’ve been very busy the last two months working on the US Social Forum. As a result - I’ve neglected liberace, my laptop. I went a full two months without even running apt-get update. Since liberace is running Lenny, the testing version of Debian, when I finally updating and then ran apt-get dist-upgrade, I had over 300 packages to upgrade, including a kernel upgrade.
I should know better, but in the interest of time I simply pulled them all in together.
When I rebooted, I got an error message:
/bin/sh: can't access tty; job control turned off

Looking up the screen, I discovered the error message that was causing the problem:
mount: Mounting /dev/hda6 on /root failed: No such device

I was fortunately dropped into an Ash shell, giving me some tools to play with.
I tried:

cat /proc/partitions

And saw all my normal partitions, including /dev/hda6, which is my root partition.
I then tried:

mount /dev/hda6 /root

And I received the message:
mount: Mounting /dev/hda6 on /root failed: Invalid argument

For kicks I tried:

mount -t ext3 /dev/hda6 /root

And presto it worked. I then manually ran:

mount /sys /root/sys
mount /proc /root/proc

Since those two commands failed before I was dropped into the ash shell.
Then, I typed:

exit

And liberace booted up without a hitch.
In an effort to trouble shoot, I re-booted liberace (and got the same  results). I also tried using previous kernels - also with the same results (so it doesn’t seem to be related to the kernel upgrade).
At one point, after getting dropped into the Ash shell, I tried:

mount -t ext3 /dev/hda6 /root
umount /root
mount /dev/hda6 /root

And that worked.
Ash doesn’t have fsck, so I re-booted into Recovery Is Possible and ran fsck on /dev/hda6. No problems.
So… I can boot if I run those manually commands. However, I have no idea why the kernel can’t mount the root partition.


First public drupal module: Registrar
Mon, 08 Jan 2007 09:32:44 +0000
Some of the folks working on the Georgia Progressive Summit (who we worked with on the USSF asked if they could use the code we developed for the registration at the USSF.
Below are my efforts at abstracting it to meet more generic requirements.
Please read the README file - it contains important information about things like integrating with the LoginToboggan module.
I’m still not sure how to publish it on drupal.org…


Freda Rosen
Mon, 08 Jan 2007 09:32:44 +0000
I have spent the last week celebrating the life of Freda Rosen, a revolutionary who passed away Tuesday morning July 17. In keeping with Freda, it was not something I did alone. We did it together - we being the powerful community Freda created and developed her entire lifetime, a community that includes, in no small part, May First/People Link.
I met Freda through MFPL co-founder Josue Guillen. Together we participated in Freda’s Social Justice Leadership Group, which met every couple months to discuss and build our leadership within our organizations. Josue, who had been meeting with Freda’s social therapy groups pushed me to join those as well, which I did a little over 2 years ago. And wow - what an impact, not only on me personally, but on the work I did everyone in my life, including May First/People Link. In particular - her insights about how groups work, how to challenge individualism, and, more than anything, how to give continue to transform my life and influence my work.
I was trying to describe that impact when I read a beautiful piece Nancy Swartz wrote on the same topic and decided to repost what she wrote instead. Thanks Nancy for such a wonderful description!

I just want to take a moment to pay homage to someone who has passed on from this earth corporeally, but who has left a strong imprint in presence, dare I say - in spirit. I hesitate to say a “in spirit,” because she’s is not a person who goes for religious symbolism or language of religion, but taken out of it’s religious context - her spirit lives on. Freda Rosen. She doesn’t have much use for religion or morality, but Freda is one of the most humane people I’ve ever met.  And if “good” weren’t such a moral term, I’d say she’s a good person. She is caring and but does not get clouded by emotions. She is so practical in her observations in a way that can be almost infuriating if you find yourself quite emotional about the topic at hand, or if the observation she makes cuts close to home. Freda has a sniper’s eye for reading a situation and knowing exactly what the power dymanic is and what is going on. She can even tell you how to manipulate it and shift the power, if that is possible. But she doesn’t always tell you that.  Freda is not interested in teaching people how to manipulate, she is interested in “development” as she put it. She wants people to grow, and be powerful and live their lives fully and engaged. Or at least that’s how I understand it. And though she does not go for the language of morality, I’ll have to say in superhero terms - she uses her powers for good and not evil.

Freda practices social therapy which has a complicated reputation due to some shady folks and self-appointed messiah-types associated with it.  But she has a real gift for social therapy, and when she went on her own to practice, many people went with her. I found Freda when I was pretty young. While at the time I had political awareness and was already a combat-boot wearing dyke performance artist who thought I was “political,” it’s Freda who really politicized me. Freda taught me the politics of everyday life. And she only ever wants me to be powerful, much as I fought her on that - wanting to have things my way and still be powerful. Sometimes the two didn’t quite mesh - having it my way and being powerful, and she tried to tell me that, even if I was a bit hard-headed about listening. I’ve become a little less hard-headed about that - thanks to Freda. She does not believe in institutions. They aren’t good for development in her view. So her therapy is very different from traditional “psychotherapy” which is quite institutional. To be a bit reductive - psychotherapy assumes that you are fucked up, and if you fix your “problems” you’ll be ok, whereas Freda’s social therapy recognizes that the world is fucked up, racist, sexist, misogynist, homophobic, anti-semitic, classist and she wants to help people be powerful given thata**s the world we live in. She’s like “ya, it’s that way, but what do you want to do given that’s the situation.” She is just practical about it. She has no use for the chip on the shoulder thing. She doesn’t judge it; she just doesn’t think it’s useful. Pro-active rather than reactive is how she encourages people to be. She has helped people transform their lives and do amazing things that will change the world for the better. She’s loves success, and though she comes from Communist roots, she is happy to see folks making money. She believes in living well as one can. I stopped going to Freda’s group a while back, but have stayed in touch and re-connected with group recently. I feel sad that Freda’s physical life is ending at a relatively young age. I guess I figured she’d be sitting in that chair on the upper west side leading group well into her 70’s. But she lives in the present and leaves her mark here and in the future. Freda recognizes that a relationship continues after someone’s death and so in honor of my relationship with her, I wanted to give a shout out to her work, a step toward keeping it alive. She’d like that.


Get lucky on Friday the 13th with the Rude Mechanical Orchestra!
Mon, 08 Jan 2007 09:32:44 +0000
The RMO contributes to the left on a near weekly basis, playing benefits, demonstrations and rallies: now we need to support them! Friday, July 13th,  2007: Caliper Studios, 67 Metropolitan Avenue, 2nd floor (between Kent and Wythe) Williamsburg, Brooklyn

Help us on our mission to cast away the bad luck of the past 8 years!!
*Friday, July 13th, 2007
*2 sets by the Rude Mechanical Orchestra, at 8 and 11pm.
*More music by Phil Not Bombs, Bovine Homecoming, and other RMO friends.
*Music for dancing by two great DJs
*Our debut CDs for sale, and other RMO goodies and surprises!
*$0-20 sliding scale, $20 gets you a free CD
*Caliper Studios, 67 Metropolitan Avenue, 2nd floor (between Kent and
Wythe) Williamsburg, Brooklyn
The Rude Mechanical Orchestra is proud to announce the release of their debut CD, featuring studio recordings of radical marching band tunes and a live performance video. This limited edition enhanced CD was created to spread the love and noise, and to raise funds for our travel goals for 2008.  Established in 2004 for the March for Women’s Lives and to protest the Republican National Convention in NYC, we hope to continue using our music and energy to support local struggles and also travel to events like the 2008 RNC (St. Paul, MN), swing-state voter registration, and brass band/movement convergences in the states and internationally.
We’ve loved supporting you over the years, and now we’d love your support!
SEE YOU IN THE STREETS!
love & noise,
the Rude Mechanical Orchestra
For the cool flyer, check out:
http://rudemechanicalorchestra.org/benefitflyerweb.jpg


Getting burned by dhclient
Mon, 08 Jan 2007 09:32:44 +0000
A member switched from a dynamic IP address to a static IP address yesterday - very exciting! And most remarkable - the switch was extremely smooth. On our debian router, I simply:


Edited /etc/network/interfaces, changing “dhcp” to “static” and added the lines in the stanza with the new static information


Restarted the interface:



ifdown wan; ifup wan

And presto it was working!
Until the next day when the Internet was down. Crap. I’m very proud of the staff person I spoke to who went through the steps of logging in and reading back the output of /sbin/ifconfig, indicating the the wan interface wasn’t assigned an IP address. She re-ran the command to bring down the interface and bring it back up and again we were back on line.
After logging in myself, I started grepping through syslog. It was really difficult because of all this dhclient noise. I started fashioning a grep command to eliminate the dhclient stuff when it dawned on me:
I never turned off the dhclient.
So, in the middle of the night, dhclient, blissfully unaware of the new static IP address, asked the world for a new IP address, didn’t get a response, and set the wan interface to not have an IP address at all.
So, for future reference, the proper order of operations should have been:

Bring down the wan interface (and dhclient with it):


ifdown wan



Edit /etc/network/interfaces to change the dhcp to static


Bring up the wan interface



ifup wan

For the curious - I killed the dhclient with:

ps aux | grep dhclient

This returns the PID as the number in the second column
Then:

kill 



Hacking Auth on MediaWiki
Mon, 08 Jan 2007 09:32:44 +0000
For some explanation of how to hack your own auth system into MediaWiki, a couple good resources are:
http://meta.wikimedia.org/wiki/User:Otheus/Auto_Login_via_REMOTE_USER
http://www.x-tend.be/~raskas/blog/2006/11/17/mediawiki-remote-user-authentication
I wanted to do someting a little different:

Rather than use HTTP Auth, I wanted to use MediaWiki’s login form and authenticate against a different database.
I also did not want any user who is not authenticated to be able to view the wiki at all.
And, I wanted users that I choose to be able to register and login properly.

Here are the steps I took - many thanks to Raska!


Copy includes/AuthPlugin.php to a different file with a new name (such as mfpl_auth_plugin.inc.php)


Edit the file
// Include the original file (change path as needed)
include (’/usr/share/mediawiki/includes/AuthPlugin.php’);
// Optionally include any libraries you might personally
// use for authentication here
// Give the class a new name and extend the original class
class mfpl_auth_plugin extends AuthPlugin {
// define a new constructor function
// this function adds a new function that is always called on
// page loads (it will be described below)
function mfpl_auth_plugin()
{
global $wgExtensionFunctions;
if (!isset($wgExtensionFunctions)) {
$wgExtensionFunctions = array();
} else if (!is_array($wgExtensionFunctions)) {
$wgExtensionFunctions = array( $wgExtensionFunctions );
}
array_push($wgExtensionFunctions, ‘Auth_remote_user_hook’);
}
// Review the remaining functions - I only modified the authentice
// function


Add a new function to the bottom of the file
function Auth_remote_user_hook()
{
global $wgUser;
global $wgRequest;

// For a few special pages, don't do anything.
$title = $wgRequest->getVal('title') ;
if ($title == 'Special:Userlogout' ||
$title == 'Special:Userlogin') {
return;
}

$wgUser = User::loadFromSession();
$username = strtolower($wgUser->getName());
global $wgAllowedUsers,$IP;
if(in_array($username,$wgAllowedUsers))
{
// Do nothing if session is valid
if ($wgUser->isLoggedIn()) {
return;
}
}

// If it is not valid log them out
include("$IP/includes/SpecialUserlogout.php");
wfSpecialUserLogout();
}

Edit your LocalSettings.php file
// Add to the bottom of the file (change path as appropriate):
require_once('/path/to/mfpl_auth_plugin.inc.php');
$wgAuth = new mfpl_auth_plugin();
$wgAllowedUsers = array(
'joe','susie','jane'
);



How do you recover a root password on a debian system
Mon, 08 Jan 2007 09:32:44 +0000
I know some people might think this is not the information that should be publicly published - but it is important to know and, if we are not encrypting our filesystems, we should acknowledge that if someone has physical access to your machine, then it can belong to them.
So having said that…
I’m assuming you are running Grub and I’m assuming that your root partition is not encrypted.
Once Grub boots, select the kernel that you want and press e to edit it.
Then scroll to the line with the kernel on it and click e again.
Then, at the end of that line, add:
init=/bin/sh

This says: after loading the kernel, directly enter a bash shell.
Then hit enter.
Then hit the b key to boot.
After the kernel boots, you should get a bash prompt. At this point, your root partition is mounted read-only - so you won’t be able to change the password on it until you make it read-write.
mount -orw,remount /
This means don’t try to mount the root partition, instead remount it with the new option: rw.
Then, change the password with:
passwd

Now, your password has been saved. Now - remount the file system as read-only:
mount -oro,remount /

Now you are back to where we started before the password change.
exec init

This means replace the current process with “init” - and that is what the kernel does by default (before we changed it to init=/bin/sh) - so the system should now boot normally.


HTML Email
Mon, 08 Jan 2007 09:32:44 +0000
I’ve never been a big fan of HTML email, however, we have received so many requests about how to do it that I’ve been re-considering. There’s a good site here that includes those reasons (it also includes a note from the author about how he is beginning to change his mind). If you are considering sending html email, please read that page first!
I asked my radical techie friends what they though and got a variety of answers, mostly relating to how to do it if you’re going to do it. Here’s a brief summary.

Keep it simple. Don’t add fancy tables and such.
Test in multiple browsers. Check out this page to get a sense of what can go wrong.
Thunderbird is the most recommended email program for creating standards compliant email. Another suggestion was to compose the email in Nvu.
Don’t expect people to see your images!! Be sure to include Alt text and use designs that won’t fall apart if the images are not viewable. Also, images as headers are not a good idea for these same reasons.
People spend on average 50 seconds reading organizational email
Sending images embedded into the email will make them more likely to be viewed (as opposed to sending links to the images). However, it will bloat your email.



Importing data into databases
Mon, 08 Jan 2007 09:32:44 +0000
I’ve spent a large chunk of my life importing data from one database into another and have only recently realized the time savings involved with scripting that work. My previous method had been to convert the original data into a format I could manipulate in a spread sheet. Then, manually tweak the data (using calculations whenever possible). Then saving the spread sheet in a format that MySQL could import.
It makes sense and it works.
The only problem is that the data entry has to freeze during this process, which can often take days. Furthermore, after spending hours doing the import, you may realize that you made mistake early on. Often, this means repeating the entire painful process.
There is a better way!
Although it takes more time up front, you will often save time by scripting the entire process. You may need one manual step to dump the data from the original database (this could be automated depending on where you get the data from). Then - write a script that parses this data, tweaks what needs to be tweaked, and then automatically imports it into the destination database.
The two advantages are:


You can work on a copy of the data all you want without having to freeze data entry. Once you have it working and you are sure the import was successful, you can do the final import with a single command.


If you make mistakes (woops, switched a column or forgot a chunk of the original data) you can easily make a small tweak to your script and re-run the whole process.


More enjoyable. Who wants to tweak data by hand when you can be programming?


An important tool in this type of endeavor is a good script. Attached to this blog is a PHP class for reading and parsing delimited or variable width data files. By default, it works pretty well on your standard tab delimited data file, but can be used for any type of delimited file or even variable width files.
The top of the class has sample php code for how to use it. If you like it, leave a comment on the blog!


Importing data into databases
Mon, 08 Jan 2007 09:32:44 +0000
I’ve spent a large chunk of my life importing data from one database into another and have only recently realized the time savings involved with scripting that work. My previous method had been to convert the original data into a format I could manipulate in a spread sheet. Then, manually tweak the data (using calculations whenever possible). Then saving the spread sheet in a format that MySQL could import.
It makes sense and it works.
The only problem is that the data entry has to freeze during this process, which can often take days. Furthermore, after spending hours doing the import, you may realize that you made mistake early on. Often, this means repeating the entire painful process.
There is a better way!
Although it takes more time up front, you will often save time by scripting the entire process. You may need one manual step to dump the data from the original database (this could be automated depending on where you get the data from). Then - write a script that parses this data, tweaks what needs to be tweaked, and then automatically imports it into the destination database.
The two advantages are:


You can work on a copy of the data all you want without having to freeze data entry. Once you have it working and you are sure the import was successful, you can do the final import with a single command.


If you make mistakes (woops, switched a column or forgot a chunk of the original data) you can easily make a small tweak to your script and re-run the whole process.


More enjoyable. Who wants to tweak data by hand when you can be programming?


An important tool in this type of endeavor is a good script. Attached to this blog is a PHP class for reading and parsing delimited or variable width data files. By default, it works pretty well on your standard tab delimited data file, but can be used for any type of delimited file or even variable width files.
The top of the class has sample php code for how to use it. If you like it, leave a comment on the blog!


Installing mod_auth_openid on Debian Etch
Mon, 08 Jan 2007 09:32:44 +0000
I followed the directions here.
This seems to work on Debian Lenny, but there is a world of unmet dependencies for Debian Etch.
apt-get update
apt-get install libopkele-dev

I also had to install a generic package not already installed on my system:
apt-get install automake

Then pull in the libdb dev package:
apt-get install libdb4.4++-dev

Then, download the source from http://butterfat.net/wiki/Projects/ModAuthOpenID/Releases.
Next run:
./configure
make
sudo make install



Internet Rights workshop software
Mon, 08 Jan 2007 09:32:44 +0000
I’m remiss!
I promised to publish the software we used during our Social Forum workshop but haven’t gotten around to it (until now).
Please see the attached zip file.
And please keep in mind: this software was written for a controlled environment - not be run over the Internet! There is no authentication (except by IP address).
If you have any questions or would like help getting it setup, please feel free to email me at jm -at- mayfirst.org.


Liberating your domain name from Tucows
Mon, 08 Jan 2007 09:32:44 +0000
You’ve probably heard us rant about hosting providers that offer to help you out by registering your domain name for you (if not, please check out Alfredo’s blog on that topic).
We increasingly find ourselves helping our members liberate their domain names from such situations. I’ve even started seeing situations when a domain expires, the next day it is paid for, yet for weeks it is still set to status “CLIENT HOLD.” It takes any number of phone calls to either the registrar, or in the worst case scenario, the registrar and the reseller.
This has got to end! Please register your domain names directly with a registrar, such as dotster.com or godaddy.com not through an intermediary!! Hopefully, some day, we’ll have a registrar with politics. But until then, at least deal directly with one corporation, not two or more when you are setting up a domain name.
Here’s some steps I find myself taking quite frequently, published for the world in the hopes that it will help others cut through the red tape more easily.
The first step is a whois query. I run Debian linux which has an easy to use whois package. If you don’t have that program and can’t install it, try using this web application instead.
You should see a line like this:
Sponsoring Registrar:Tucows Inc. (R11-LROR)
If it says Tucows you’re probably in trouble. They work with a lot of resellers. So many in fact, that they have their own special whois program to tell you which reseller controls your domain.
At the bottom of the page you should see something like this:
Registration Service Provider:
easyDNS Technologies Inc., easydns@myprivacy.ca
+1.416.535.8672
http://www.easydns.com
This company may be contacted for domain login/passwords,
DNS/Nameserver changes, and general domain support questions.
If the number actually works and the organization is still in business and cares about who you are, you’re in luck. Otherwise, you’ll need to follow up with Tucows to get whatever registration issues resolved.
Once you do get them resolved, don’t stop! Instead, hound them to give a Transfer Authorization code that will allow you to move your domain to another registrar.


Macintosh + Firefox  + WMV + AutoStart false = AutoStart true
Mon, 08 Jan 2007 09:32:44 +0000
Coding for multiple browsers and operating systems is always painful. However, adding wmv to the mix makes it even worse.
After loads of testing, we managed to get this code to work on Firefox for Windows, IE6 and IE7 for Windows, and Firefox/Iceweasel on Linux.
Now, we’ve discovered that with Firefox (and Safari) on a Mac, the AutoStart=false is ignored - the video starts playing automatically.
&lt;object id="MediaPlayer"
classid="CLSID:22D6F312-B0F6-11D0-94AB-0080C74C7E95"
standby="Loading Microsoft Windows Media Player components..."
type="application/x-oleobject"&gt;
&lt;param name="FileName" value =
"http://www.afterdowningstreet.org/downloads/norman.wmv"&gt;
&lt;param name="AutoStart" value="0"&gt;
&lt;param name="ShowControls" value="true"&gt;
&lt;param name="ShowStatusBar" value="true"&gt;
&lt;embed type="application/x-mplayer2" Name="MediaPlayer"
src="http://www.afterdowningstreet.org/downloads/norman.wmv"
AutoStart="false" autoplay="false" ShowStatusBar="1"
ShowControls="1" volume="-1"&gt;&lt;/embed&gt;
&lt;/object&gt;

Looking at this post I get the impression that autostart is not supported on a mac.


May First Day of Maintenance: Saturday, September 22
Mon, 08 Jan 2007 09:32:44 +0000
Hi May First/People Link folks! Today (Saturday September 22) we are doing a major series of upgrades on all of our servers which may result in brief periods of downtime (see our service advisory). If you are having trouble connecting, please be patient while we complete the upgrades. If you are experiencing problems and it is after 9:00 PM Eastern/New York time, please email us at support@mayfirst.org. Thanks!
Click the more link to see our progress reports as we go!
We’re starting with leslie, our email list server (aka lists.mayfirst.org). We’ve just starting installing the new packages as we upgrade from Debian Sarge to Debian Etch. Once all packages are installed we will reboot.
Update: having some troubles with upgrading mailman - the list server program. Trouble shooting is time consuming, so we’re going to upgrade chavez (which this blog is hosted on) while we wait.
Leslie is done!!
Chavez is now done too!!
Malcolm is done
Robeson is done
Shadow is mostly done, but the Tachanka collective will need to put some more work into it
Octavia (hosting rosa) is done
Gramsci (hosting mendes and peltier) is done too!
Sontag (hosting moses and assata)
Menchu is done
Unfortunately - we didn’t finish all the servers. Still to do:
octopus (serial console installation)
evo (sarge -> etch)
viewsic (sarge -> etch)
chun (sarge -> etch)
harry (move to domU)


May First Maintenance Day Continues: Sunday, September 30
Mon, 08 Jan 2007 09:32:44 +0000
We’re continue our day of maintenance at May First/People Link. Click the more link below to say abreast of how things are going.
During the course of the maintenance we’ll be working on the following machines:
Octavia (running our onsite backup)
Viewsic (shared hosting server)
Evo
Chun (Delaware server)
Harry (Members control panel server).
We’ll keep you posted (on this blog) of our progress.
Octavia has been re-booted and now has an “Octopus” installed. The octopus gives us “serial” access to our other servers. This means that if one of our other servers goes down, we can still connect to it via the serial access on octavia. Now we are working on configuring the octopus.
The octopus is now setup! We’re done with Octavia.


More etch upgrades: Peewee
Mon, 08 Jan 2007 09:32:44 +0000
And another etch upgrade as Debian Etch approaches stable! For this server, having downtime happen earlier (before people come into the office) was the priority.
Step 0: Edited my /etc/apt/sources.list, changing sarge to etch (as an aside, not sure why anyone would use “stable” in their sources.list - seems like an invitation to upgrade a distribution without meaning to).
Step 1: Upgrade the kernel

apt-get install linux-image-2.6-686

Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
busybox initramfs-tools klibc-utils libc6 libc6-dev libklibc libselinux1
libsepol1 libvolume-id0 linux-image-2.6.18-4-686 locales lsb-base makedev
module-init-tools tzdata udev
Suggested packages:
glibc-doc manpages-dev linux-doc-2.6.18
Recommended packages:
libc6-i686
The following packages will be REMOVED:
base-config hotplug initrd-tools kernel-image-2.6.8-2-386
The following NEW packages will be installed:
busybox initramfs-tools klibc-utils libklibc libselinux1 libsepol1
libvolume-id0 linux-image-2.6-686 linux-image-2.6.18-4-686 lsb-base tzdata
udev
The following packages will be upgraded:
libc6 libc6-dev locales makedev module-init-tools
5 upgraded, 12 newly installed, 4 to remove and 238 not upgraded.
Need to get 29.3MB of archives.
After unpacking 11.3MB of additional disk space will be used.
Do you want to continue? [Y/n]

Ok, sounds good.
But wait…
Removing kernel-image-2.6.8-2-386 ...

You are running a kernel (version 2.6.8-2-386) and attempting to remove
the same version. This is a potentially disastrous action. Not only
will /boot/vmlinuz-2.6.8-2-386 be removed, making it impossible to boot
it, (you will have to take action to change your boot loader to boot
a new kernel), it will also remove all modules under the directory
/lib/modules/2.6.8-2-386. Just having a copy of the kernel image is not
enough, you will have to replace the modules too.

I repeat, this is very dangerous. If at all in doubt, answer
no. If you know exactly what you are doing, and are prepared to
hose your system, then answer Yes.
Remove the running kernel image (not recommended) [No]?

I guess I missed that part about removing the kernel. Why does it need to remove the existing kernel??? Ah - it’s trying to remove initrd-tools. Well, I said No to the prompt above (I guess I had no choice).
Then I ran:

apt-get -f install

And got:
mayfirst@peewee:~$ sudo apt-get -f install
Reading Package Lists... Done
Building Dependency Tree... Done
Correcting dependencies... Done
The following extra packages will be installed:
initrd-tools libc6 libc6-dev libdevmapper1.02 libselinux1 libsepol1 locales
tzdata
Suggested packages:
glibc-doc manpages-dev
The following NEW packages will be installed:
initrd-tools libdevmapper1.02 libselinux1 libsepol1 tzdata
The following packages will be upgraded:
libc6 libc6-dev locales
3 upgraded, 5 newly installed, 0 to remove and 240 not upgraded.
Need to get 70.7kB/12.0MB of archives.
After unpacking 3568kB of additional disk space will be used.
Do you want to continue? [Y/n]

Ok, sounds good. I know I said that before when it really didn’t sound good, but this time I meant it.
So then I tried (again):

mayfirst@peewee:~$ sudo apt-get install linux-image-2.6-686

And got:
mayfirst@peewee:~$ sudo apt-get install linux-image-2.6-686
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
busybox initramfs-tools klibc-utils libklibc libvolume-id0
linux-image-2.6.18-4-686 lsb-base makedev module-init-tools udev
Suggested packages:
linux-doc-2.6.18
Recommended packages:
libc6-i686
The following NEW packages will be installed:
busybox initramfs-tools klibc-utils libklibc libvolume-id0
linux-image-2.6-686 linux-image-2.6.18-4-686 lsb-base udev
The following packages will be upgraded:
makedev module-init-tools
2 upgraded, 9 newly installed, 0 to remove and 238 not upgraded.
Need to get 0B/17.3MB of archives.
After unpacking 52.4MB of additional disk space will be used.
Do you want to continue? [Y/n]

Oh good. No kernels being removed.
Ack - I wish I had added –purge to my original apt get command so that hotplug could have been purged. Ok, next time.
Step 2: reboot, to get that out of the way early on
Wow, it comes right back up. I guess I’ll always be impressed when that happens.
Step 3: Upgrade Samba (after which people will be able to login)

apt-get install samba

Which returns:
mayfirst@peewee:~$ sudo apt-get install samba
Password:
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
libcupsys2 libgcrypt11 libgnutls13 libgpg-error0 libkrb53 libncurses5
libopencdk8 libpopt0 libreadline5 libtasn1-3 readline-common samba-common
smbclient
Suggested packages:
cupsys-common rng-tools gnutls-bin krb5-doc krb5-user samba-doc smbfs
Recommended packages:
libtasn1-3-bin smbldap-tools
The following packages will be REMOVED:
libcupsys2-gnutls10
The following NEW packages will be installed:
libcupsys2 libgnutls13 libreadline5 libtasn1-3 readline-common
The following packages will be upgraded:
libgcrypt11 libgpg-error0 libkrb53 libncurses5 libopencdk8 libpopt0 samba
samba-common smbclient
9 upgraded, 5 newly installed, 1 to remove and 228 not upgraded.
Need to get 11.2MB of archives.
After unpacking 7197kB of additional disk space will be used.
Do you want to continue? [Y/n]

Oh dear. Looks like a samba configuration change:
Chaining passdb backends is not supported
Beginning with version 3.0.23, samba no longer supports chaining
multiple backends in the "passdb backend" parameter.  It appears that
your smb.conf file contains a passdb backend parameter consisting of a
list of backends.  The new version of samba will not work until you
correct this.

I had no idea I was doing something as unsupported as chaining backends.
My smb.conf file contains:
passdb backend = tdbsam

Which looks to me like jut one backend. I think I should file this as a bug.
And, samba seems to be functioning properly.
Step 4: Upgrade the rest

sudo apt-get dist-upgrade



Moving Drupal Between Servers
Mon, 08 Jan 2007 09:32:44 +0000
Drupal is not so easy to move from one server to another. Sometimes it works, sometimes it doesn’t. I move a site from a server running MySQL 4.1 to MySQL 5.1 today and when I hit the page, I got a completely blank screen.
Hm.
I added set_error(E_ALL) to the top of my settings file and got a bunch of missing index messages (which apparently is normal) and then a dump of binary looking data.
After inserting some echo statements throughout the code, I realized the problem was with the cache.
DELETE FROM cache

Now it works.


MySQL bin log: Your friend
Mon, 08 Jan 2007 09:32:44 +0000
A big thanks to the Debian MySQL server maintainers for enabling bin logging by default.
I just did a big boo boo: dropping the wrong database without a backup. I don’t recommend doing this. But - if you do this, and you are running Debian with a stock configuration, there is hope.
In /var/log/mysql you should see a bunch of files named mysql-bin-00001. If you use the mysqlbinlog program you can convert those files into sql statements. So, you can do:
mysqlbinlog mysql-bin* > dump.sql

Now - you have a giant sql dump. If you want to restore from your mistake you can do something fancy awk’ing - or, if you’re like us, you can open the giant file in vim and by hand edit out the sql statements you don’t need. We did that by searching in vim with:
/^use db-name
To find all sql statements concerning the db that I dumped (lines that start with use db-name with db-name replace with the name of the database). And:
/^use
To find the next set of statements. Then we deleted all statements not related to the db I dropped. We also deleted all the comments (starting with /*) and all the statements that seemed bin log specified, like ones starting with ROLLBACK and setting the timestamp (SET TIMESTAMP).
And, at long last, we had a sql file that, when dumped into the original database, brought the lost Drupal site back to life. Woohoo.
Thanks dkg and the rest of the MFPL tech crew for the help!


Nagios2 and complex configurations...
Mon, 08 Jan 2007 09:32:44 +0000
I have a love/hate relationship with Nagios. On the one hand, I love how much power there is in the object oriented approach to the configuration files. On the other hand, it takes a long time to figure out what’s going on and how to fix them.
I wish there was a nagios command that would print out exactly how each service, host, etc. is inheriting. Like the postconf command which spits out all your configuration variables, including the default ones, that are currently active.
In any event…
The goal of my latest change was to stop nagios from bothering us every time a packet was delayed in our backup data center. Because we run a bunch of high volume backups at the same time in the night, we get lots of delayed packets.
All of our servers use a generic template with the check_command set to check-host-alive, which will notify us if the host is outright dead.
In addition, we have a service, in which all servers are a member, called ping-servers. This service checks for ping latency. I’ve now turned notification off for this service, so I don’t get woken up in the middle of the night - yet we still get to see a report on latency when logging into nagios.
This now applies to all of our servers. I think we’ll be in a stronger position if we only get notifications for real emergencies rather than a high volume of notifications for delayed packets, which is good to know (and Nagios will keep track for us) but does not always require immediate action on our part.


Naming your network connections
Mon, 08 Jan 2007 09:32:44 +0000
Having your network connections called eth0 and eth1 is fine if you are a machine. However, for us humans, it’s nice to give them better names. For example, on my laptop, I’ve named my interfaces “wireless” and “cable.”
To do that (assuming you have a modern kernel and assuming you are running Debian), you should edit:
/etc/udev/rules.d/z25_persistent-net.rules

Other distro’s may have to look elsewhere.
After doing that - rebooting your computer will cause the new names to take effect.
However, if you don’t want to reboot, you can force the driver to unbind from the interface and rebind by following the directions here: http://lwn.net/Articles/143397/.


New blog
Mon, 08 Jan 2007 09:32:44 +0000
I’ll be continuing to post both general interest and tech-related blogs here. However, I’m beta testing our new WordPressMU installation (blogs.mayfirst.org) with a new blog that is dedicated to profiling May First/People Link members. Please check it out here: blogs.mayfirst.org/jamie.


New Drupal Tutorial
Mon, 08 Jan 2007 09:32:44 +0000
Beginning drupalists will be interested in our new Drupal tutorial, written by Amanda B and available here.


Next Generation Email list software
Mon, 08 Jan 2007 09:32:44 +0000
Mailman and Sympa are two very impressive free email list programs. I’m especially fond of Mailman, which has done a superb job powering Mayfirst/People Link email lists for years and years.
However, IMHO, the world is ready for the next generation email list. When I think of the needs of MFPL and the progressive movement, and I consider the number of activists with beginning php skills and the varied ecosystem of databases and communications programs written in PHP, it really points to a need for a fully modular, library-based, PHP based email system.
In other words, we don’t need another monolithic program. I’d rather see a suite of libraries that do all the work and a simple, light-weight example implementation of that library. This way, the core work can happen on the libraries, and anyone in the world can tie the libraries together to meet their needs.
Here is my first stab at the libraries that are needed. Many of the libraries are already written!
##Classes##
Logger: logs all messages, errors or otherwise
Authenticater: the person with this username really is the owner o that user name.
Authorizer: the person with this username can do x, y, z.
Bouncer: takes a bounced message, extracts the list it was sent to, the address it was sent to, and whether or not it is a hard or soft bounce
BounceProcessor: takes an email address, a list name, and hard or soft bounce and records that information for the given address.
BounceManager: iterates through each list, reads the particular bounce processing rules for the given lists (unsubscribe after X bounces, etc.) and takes the required action.
UserManager: takes a single user and gets/sets subscriber settings, like digest, vacation, unsubscribe, change password, get lists they are subscribed to, etc.
SubscriptionManager: takes a given list and generates a list of emails with their preferences for the given list or adds/removes subscribers from a list.
ListManager: takes a single list and gets/sets configuration information, subject line tag, etc. The subscription object is a property of the list manager object, which enables the list object to know who is subscribed to it.
Generator: takes a ListManager object and generates every customized message. These are then handed to the injector.
Injector: takes a message and injects it into the mail system
Archiver: takes the name of a list and generates and pages and show search results for a given list archive.
Hasher: takes a list message and generates a secret hash (and stores it). Then takes each email address and list name and creates unique hash to be embedded in the message that a user can click on to unsubscribe from the list.
Implementations
Here’s a simple description of how a message would be sent using these libraries:
$message = "The email to send.";
$list_name = "thelist";

// create all the classes we'll want to use
$logger = new Logger();
$subscription_manager = new SubscriptionManager();
$list_manager = new ListManager();
$generator = new Generator();
$injector = new Injector();
$hasher = new Hasher();

// Each class should get the logger
$subscription_manager->set_logger($logger);
$list_manager->set_logger($logger);
$injector->set_logger($logger);
$generator->set_logger($logger);
$hasher->set_logger($logger);

// Initialize the subscription class with the
// list we're using
if(!$subscription_manager->set_list_name($list_name)) return false;

// Initialize the list class with the
// list we're using
if(!$list_manager->set_list_name($list_name)) return false;

// Now pass the subscription class on to the list class
$list_manager->set_subscription($subscription_manager);

// OK - now the list class knows all of it's configurations and
// because it has a subscription class, it knows how to get
// all the email addresses (and particular settings for those
// email addresses).

// Now pass the list class on to the generator class, which is going
// to generate all the individual messages
$generator->set_list_manager($list_manager);

// Give the generator the hasher so it can append the unique hashes
$generator->set_hasher($hasher);

// Now tell the generator object what message to send
$generator->set_message($message);

// The generator's job is only to generate individual emails - not to inject
// them into the mail system. That's the job of the injector. So - pass the
// generator the injector class so it can use the class to actually inject
// the messages.

$generator->set_injector($injector);

// Now tell the generator to send away
if(!$generator->send()) return false;

return true;

And bounces would be managed by an address that pipes them to a program like this:
// Dealing with bounces

// $message should be filled with standard in

$bouncer = new Bouncer();
$bounce_processor = new BounceProcessor();
$bounce_processor->set_bouncer($bouncer);
$bounce_processor->set_message($message);



Nodaddy: the Internet does have a hierarchical and centralized aspect
Mon, 08 Jan 2007 09:32:44 +0000
At our radical techie brunch last weekend, I heard about NoDaddy, a site documenting how someone’s entire domain name was pulled from the Internet by GoDaddy
For those of you unfamiliar with the technology: when you ask the Internet to look up a website for you, there is one single authority for each “top level” domain. In other words, if your domain ends in “.org” the authority is Public Interest Registry. Public Interest Registry is in charge of telling the world how to find the computer that corresponds to the domain name you are looking for.
Public Interest Registry, in turn, contracts out with a few dozen companies (like GoDaddy, Network Solutions, Dotster, etc.) who handle the business end of the transaction. You pay them money, and they take care of inserting your domain into the Public Interest Registry database.
Keep in mind: this has almost nothing to do with the organization that is hosting your web site.
In other words, suppose your web site (say mayfirst.org) is hosted with us. And suppose it’s politically sensitive and might be considered a threat to a government, so you have copies on many different servers with many different hosting providers in many different countries (maybe mirror1.mayfirst.org, mirror2.mayfirst.org, etc). Now, if one server gets confiscated or shut down, you have many other servers. No problem.
However, if your domain name registrar pulls your domain name (as is documented on NoDaddy), you really are in trouble because nobody will be able to find the server that has a backup mirror copy of your site. In other words all .org Internet traffic, at some point, goes through Public Interest Registry before it goes to your hosting provider.
At the radical tech brunch we discussed advertising “tags” instead of domains as an alternate. For example, rather than saying: go to mayfirst.org, we should be saying: go to your favorite search engine and search for MayfirstCampaignOnX or something like that.
Then, we can put up information on many different computers with, most importantly, different addresses.


One step forward, two steps back
Mon, 08 Jan 2007 09:32:44 +0000
I was excited when a colleague forwarded to me an email about a new journal called Journal of Information Technology in Social Change. This is exactly what we need, and it’s being headed up by Michael Gilbert, who I know first hand has a lot of experience in the world of social justice and technology. Looking through the titles of the articles interested me and made me want to read more. But wait, where are the links?
The Journal of Information Technology is not available on line. Or, it is available but by subscription only. I can understand wanting to re-coup from the expense of publishing - I know that’s expensive and don’t expect people to work for free. On the other hand, there’s no indication of whether issues will become available at a later date. Will they ever be available for the public? I really hope so!!
Looking further, I saw the copyright and subscription notice:

Personal License:

If you have purchased a copy/subscription to the Journal with a personal license, this means that it is for your personal use. You may make copies for backup purposes or to allow you to personally use this report on more than one computer. You may also print copies, but not for circulation of any kind.

Corporate License:

For most of you, we recommend a corporate license. If you have purchased a copy/subscription to the Journal with a corporate license, this means that it is for use by people within your organization. You may make paper copies for internal circulation. You may post it to your intranet, so long as access to that intranet is restricted to those who work for your organization.
Yeesh. This sounds like the ravings of the music industry execs who have no understanding of the Internet.
I don’t want to over simplify the situation: just making everything free is not a viable solution in all cases. We need to pay for labor!
On the other hand, how can we use licensing to build community? Quite honestly, and maybe just for childish reasons, whenever I’m debugging someone’s Windows computer and that pop up asks me if I want to send a bug report to Microsoft I say no. I don’t want to contribute to a corporate software community. On the other hand, I do, whenever possible, seek out the issue tracking system for any free software that I use. Why? Because it’s about more than getting good software - it’s about developing and building communities around that free software.
The person who forwarded the email about this journal suggested that we submit a piece about our experiences working on the Social Forum. I’m sure, given the subscription system in place, that if accepted we would be paid for the piece. But, given my time and politics, I’d contribute for free to a project with the primary purpose of sharing information then contribute for money to a project that a subscriber can’t even forward to their mother!


OpenID and Trac
Mon, 08 Jan 2007 09:32:44 +0000
With OpenID installed, I wanted to get the OpenID trac module working.
I started by installing some prerequisites:
apt-get install python-setuptools python-crypto python-elementtree

I then downloaded some python software not in the Debian Repository:


python-yadis / http://www.openidenabled.com/resources/downloads/python-openid/python-yadis-1.1.0.tar.gz


python-openid / http://www.openidenabled.com/resources/downloads/python-openid/python-openid-1.1.0.tar.gz


python-urljr / http://www.openidenabled.com/resources/downloads/python-openid/python-urljr-1.0.0.tar.gz


For each of these, I untarred them and then ran:
python setup.py install

Sadly it installed them into the /usr/lib/python directories (rather than /usr/local/lib). Probably a way to get that to work, but I don’t know enough about Python to really know where to start.
Next I added the following to my apache installation:
&lt;Location "/"&gt;
SetEnv PYTHON_EGG_CACHE "/srv/trac/support/egg_cache"
&lt;/Location&gt;

I created the directory specified and made sure the web server had write access to it.
Then, I downloaded the Trac OpenID plugin here: http://trac-hacks.org/wiki/OpenidPlugin
After downloading, I untarred it, entered the directory and ran:
python setup.py bdist_egg

And then copied the resulting .egg file (which was put in the dist directory) into /srv/trac/support/modules
Finally I added the following to my trac.ini file:
[components]
trac.web.auth.* = disabled
openidauth.* = enabled

And I commented out the “Location /projectname/login” lines in my apache auth file.
And restarted Apache.
Well, it’s almost there. When I enter my OpenID login I get:
Traceback (most recent call last):
File "/var/lib/python-support/python2.4/trac/web/main.py", line 387, in dispatch_request
dispatcher.dispatch(req)
File "/var/lib/python-support/python2.4/trac/web/main.py", line 237, in dispatch
resp = chosen_handler.process_request(req)
File "build/bdist.linux-i686/egg/openidauth/auth.py", line 162, in process_request
File "build/bdist.linux-i686/egg/openidauth/auth.py", line 203, in _start_login
File "build/bdist.linux-i686/egg/openidauth/auth.py", line 199, in _getConsumer
AttributeError: 'module' object has no attribute 'Consumer'

To be continued…


Organizing mass forums
Mon, 08 Jan 2007 09:32:44 +0000
It’s been a few months since the US Social Forum - a few months that I’ve spent reflecting on how things went and what I would like to see happen differently in the future.
With the US Social Forum pulling in 10,000 people and world social forums bringing in hundreds of thousands of people, we are experiencing an organizing effort unparalleled on the left in history. An important enabler of this phenomenon is the Internet, which has been both critical and under-utilized. In particular, the Internet provides opportunities to inject decentralization, freedom and transparency in our organizing strategy - yet these opportunities have only been minimally tapped.
Below are few general ideas followed by a few very specific proposals for how to organize a future gathering that might more effectively use the Internet to its fullest democratic potential .
General thoughts


No technology teams - the first mistake we can make when organizing a mass forum is to create a technology team. Most technology teams are formed with the best intentions: either technologists wanting to band together and support the organizing effort or non-technology organizers recognizing the value of technology and moving to create a space for that work to be done. The effect, however, is the same: all other organizing efforts become divorced from technology. Outreach and communications teams are formed without anyone with web development expertise, fundraising teams are formed without any database designers, logistics teams are formed without anyone who can setup a computer network. Meanwhile, a group of technologists work together to meet these needs without knowing what the needs actually are. Forming technology teams is akin to forming an “organizing” team rather than recognizing that every team is an organizing team.


Distribute technology - If one organization or one team is handling the outreach, communications, and fundraising, it would be considered bad organizing. The power of any organizing process is to involve a broad group of people. Technology is no different. And, technology lends itself to being distributed. If we think of technology not as an abstract thing that someone just has to do, but instead as an integral part of every team doing work, it becomes easier to see how it can be distributed. There is no reason why technology services like email, mailing lists, web sites, domain name service and others cannot be provided by a collection of organizations and individuals; and there’s no reason why these services cannot be run by the committees relying most heavily on them.


Organize, don’t contract - many efforts at organizing mass forums contract out the technology work to an individual or a consulting firm. This is true whether that individual or firm is being paid or doing the work pro bono (the exchange of money is not the problem - it is the contractual relationship that is problematic). This approach could be applied to any aspect of the organizing effort: outreach, fundraising, communications, etc. But we don’t do that because the act of organizing these teams builds the forum itself. Why should technology be different? Is it because technology is considered more important than these other tasks? Or because it is considered less important? A successful organizing effort should consider all labor to be an opportunity for organizing. No task should be turned over to a consulting firm, particularly a task that is integral to the organizing effort.


Transparency - everything that is done on the Internet can be documented fully on the Internet. Most of it is not documented because people don't realize that it's possible. For example, any email list archive can be made public with the click of a mouse. Most forms of Internet communication, whether it is email or web based, can be done in a way that creates a trail of replies accessible via the Internet.
All of these general ideas lead to the same place: we should place the technology used to drive organizing efforts (and the technologists to build it) with the teams that need it.
And, to take this one step further, teams should form around the technologies that will drive them, not think of technology as something to add on. If we consider how mass organizing takes place - almost every organizing activity happens either entirely via the Internet (direct email, web site publishing, databases, voip phone calls) or indirectly related to the Internet (posters downloaded and printed, phone lists downloaded and called, etc.). If we try to plan our stratey first and then figure out how technology can support it we are putting the cart before the horse.
A Model for organizing
This organizing strategy thinks first about the Internet and the technology we use to organize, and forms teams around that infrastructure. It also strives above all to achieve:

Transparency: every piece of communication and decision should be publicly available for review (and reconsideration)

Flat hierarchy - it's impossible to achieve a purely flat hierarchy, however, all efforts should be made to flatten the power structure and where that is not feasible, clearly define where the power lies.
There are two types of teams: primary and secondary. The names simply mean that the primary teams must be formed first because the secondary teams will rely on them. It doesn’t mean that primary teams are more essential or important - only a question of timing.
For the purposes of this document, I’m going to use the domain “wsf.org” as the example Internet address.
Primary teams
Identity team
The identity team is responsible for ensuring that everyone who wants to participate in the forum can get a username and a means for asserting ownership of that username (for example, with a password). In addition, this team is responsible for recording the minimum amount of private information being collected for all participants (like email address, first name, last name, etc). The identity team helps users with problems like resetting passwords, locating forgotten usernames, etc. The identity team is also in charge of writing and maintaining a privacy policy for the entire forum organizing effort.
The identity team is in charge of the server: identity.wsf.org. This server allows anyone to register their username. In addition, it handles all authentication requests for every other server being used. In other words, you only get one username and password here. Then, when you submit a workshop proposal, or create a blog, or register for the forum itself, you always enter this username and password and you’re unique identity is recorded in all places.
Backup team
The backup team is a mostly technical team in charge of maintaining backups of all data and applications in use. By having an independent backup team, if another team falls down on their job and their technical tasks need to be re-assigned, all necessary data is available to make that re-assignment. In other words, no team can use their access of data or technology to hold the organizing effort hostage.
The backup team would maintain two servers: backup.wsf.org (for data) and repo.wsf.org (for custom programming code written for the event).
Communications and Collaboration team
The Communications and Collaboration team is one of the most critical teams in the organization effort. It’s primary role is to act as the “glue” team - responsible for ensuring that all teams are communicating effectively with each other and collaborating.
Some of the responsibilities of the communications and collabration team include: scheduling regular report back and questions sessions between teams, following the progress of all teams and acting as liaisons where necessary, maintaining an issue tracking web site (inquiries.wsf.org) where anyone can post a question, problem, complaint (ranging from “What should I wear at the forum?” to “The web site is broken” and other technical requests), and ensuring that teams are receiving and responding to these requests in a timely manner.
This team’s additional technical responsibilities include maintaining and creating email lists for organizers (lists.wsf.org), maintaining a chat system (chat.wsf.org), and any other form of communications needed by the organizing effort (VOIP phone, SMS, etc.)
Translation team
This team is responsible for translation, both for Internet applications and for the event itself. This team is responsible for translation.wsf.org, which will act as a central translation server for all the other servers. In other words, all participating servers should be using a standard set of libraries that share strings to be translated across the network with this central server, allowing the translation team to focus on one database of strings to translate.
Server assignment team
This team is responsible for assigning computers to the various web addresses in use. In other words, they control what computer responds to backup.wsf.org, inquiries.wsf.org, etc. Rather than share this power with a lot of people (which would create security concerns), the team keeps tight control but only makes changes based on requests to inquiries.wsf.org. They remain responsible for making fair decisions based on feedback posted to inquiries.wsf.org. This team is also responsible for writing a policy on how such decisions are made.
Domain team
This team is responsible for the domain used for the organizing effort. The Domain team controls which name server is the authority for wsf.org, and thus can take away the power of the server assignment team. This team is the ultimate authority of the entire process, since, by changing authoritative name servers, this team can effectively hijack the entire process. This is the only team without appeal.
Secondary teams
Secondary teams will form based on the details of the event itself. However, common teams will include:
Registration Team
The registration team is responsible for collecting registration fees from participants, distributing information about programming and other notices about the forum itself, and processing registrants as they arrive. The registration team is responsible for maintaining register.wsf.org.
Logistics team
The logistics team is responsible for securing space and resources in the city where the event is taking place. These responsibilities include identifying and negotiating spaces available for workshops (and publishing them at space.wsf.org in both human-readable and machine-readable formats), setting up housing and ride sharing systems, as well as securing Internet access where needed and public access computer terminals.
Programming team
The programming team is responsible for soliciting workshop proposals, encouraging collaboration between groups suggesting similar proposals, rejecting spam and other submissions not within the broad guidelines and maintaining programs.wsf.org in a way the makes public all submissions and the decisions taken on each one. The team is responsible for publishing this information both in an easy to ready and search format for humans as well as a machine-readable format (such as RSS) on programs.wsf.org.
Scheduling Team
The scheduling team is in charge of matching available spaces (secured by the Logistics team) with the submitted workshop proposals (from the programming team). This team is responsible for publicizing the workshop schedule in a variety of human and machine-readable formats (schedule.wsf.org).
Outreach Team
This team is in charge of getting the word out about the forum to everyone who may participate in it (participation does not necessarily mean attend). This team is also responsible for language and materials used in the organizing of the forum. The communications and outreach team is responsible for www.wsf.org and blogs.wsf.org, among other potential sites.
Fundraising Team
Fundraising is responsible for the grant writing and solicitation of individual donations. Responsibilities include providing easy to use donation forms and maintaining records at donate.wsf.org.
Accounting Team
Accounting, like technology, should not be considered a “non organizing” activity. Accounting is particularly important task to be done consistently with the politics of the event: transparency is critical when it comes to managing the money. This team is in charge of accounting.wsf.org - a site publishing all transactions made for the forum.
Volunteer Team
Volunteer recruitment and assigning volunteers to teams is the responsibility of the volunteer team. This team will need to maintain a database for registering and assigning volunteers at volunteer.wsf.org.
Other teams
Not every event may organize these teams and may organize completely different teams - the list above is only a starting point intended to demonstrate the combination of technology and organizing needed to accomplish the work.
Requirements for hosting a server
Participation in any team would be similar to participation in any type of organizing committee: anyone who wants to contribute to the effort is welcome.
However, individuals or organizations that would like to host parts of the technology would need to follow several guidelines before being assigned a server:


Use of free software. This requirement is necessary to ensure that no organization can hijack the technology by creating a reliance on software that cannot be easily replicated. By using all free software we reduce our reliance on any single technology provider. Exceptions can be made (provided they are discussed on inquiries.wsf.org) - this is not intended to be a point of inflexibility - instead it is intended as a measure of security and freedom.


All data and custom code must be backed up to the project backup servers (for similar reasons as those stated in the free software guideline). By using free software and backing up all data, any service can be easily recreated on any server.


Participation in the communication channels. By offering a service for the organizing effort, the organization or individual is agreeing to be responsive to any inquiries related to that service posted to inquiries.wsf.org and to participate in regular meetings.


Use the agreed upon authentication system that utilizes the central identity team's server. This requirement is crucial - to ensure that we can identity the same user login on every server.
Power
Where does power lie in this setup?
The first place it resides is with everyone who gets an identity and posts to any of the web sites or systems allowing comments. In particular, the inquiries.wsf.org site is where decision are made. People who take the time to read and post on that site will have a greater influence on the direction of the forum organizing. People who are uncomfortable or un-knowledgeable about using these types of technologies will be left behind. It’s the responsibility of all participants to help everyone who wants to participate learn to navigate the online tools to enable their full participation.
The second place is with the teams themselves. There are no hard and fast rules about how decision making happens in these teams - however, it is within these teams that important decisions about both the organizing and the technology are to be made.
The third place is with the technology groups and individuals who maintain the servers. While they have the responsibility to make decisions in accordance with their teams, they have the power to make unilateral decisions about the direction of the technology that have the potential for making an huge impact on the forum (for example, they could remove posts, delete email lists, etc.).
The fourth place is with the server assignment team. This team is governed by the comments at inquiries.wsf.org, however, they hold the responsibility to translate the comments at inquiries.wsf.org into a decision to take services away from technology groups and individuals who are deemed to not be following the will of the participants posting on inquiries.wsf.org.
The fifth and final position of power is the domain team. This teams has the power to unseat the server assignment team by directing the domain name itself to a different team. The domain team is the final arbiter.
Although this is far from a flattened hierarchy, it has the advantaged of:

Having a clear line of power

Having a mechanism to make public all decisions and allow feedback and review of all decisions.
Next Steps
As a technology-driven approach, this proposal will require non-technologists to work hard to keep on top of the tools being used. However, it will require technologists to do something that may be even more difficult: step up as organizers and leaders. Far too many technologists covet the role of behind-the-scenes mechanic - avoiding the messiness, unpredictability and emotionality of organizing. However, if we are to make the next step in global organizing, we’ll need to develop a new generation of leaders with both technology and organizing skills.


Our web launched!
Mon, 08 Jan 2007 09:32:44 +0000
Freespeech TV just launched their amazing Ourweb campaign. If you have 4 minutes, please check out this site, the video, and tell your friends about it. I honestly think this is one of the most important campaigns on the Internet and we should fully put our support behind it.

The full press release is below:
Free Speech TV Launches “It’s Our Web” Campaign
FOR IMMEDIATE RELEASE
November 27, 2007
Today Free Speech TV launched a public campaign urging Internet users to remember it’s our web, and not the property of large corporations.  Find the launch page at http://freespeech.org/ourweb
Free Speech TV is using the “It’s Our Web” Campaign to educate the public on the dangers of an Internet subjugated by a cartel of domineering big media corporations. Free Speech TV is also using the campaign to launch its new non-profit, non-commercial open-source based social networking community that can be found at http://community.freespeech.org
The video touches on issues such as online privacy/tracking, centralized ownership, and copyright, which have been hitting the headlines lately due to new incursions by big online media conglomerates including the widely distained “Facebook Beacon”.
A growing group of bloggers, independent media organizations, open-sourcedevelopers, and Internet users have latched onto the video as a rallying cry for an Internet free of invasive marketing and advertising.
Free Speech TV Director of Outreach Jason McKain said today:
“It’s exciting to see Internet users from all walks of life sharing and
engaging with this exciting video and campaign. The popularity of video tells us that there is a prevalent underlying concern with these issues, and we are happy to provide an alternative for those who choose to get off websites geared towards manipulating and exploiting its users.”
Steve Anderson the Executive Producer of the “It’s Our Web” video said
today:
“Suddenly it feels like the Internet is being shaped to satisfy the demands of advertisers and marketers rather then us, the users. We need to embraces Internet sites, services and tools that are more independent of this excessive and unnecessary obstruction to our online travels. I hope this video serves as a tool for those demanding an Internet independent of manipulation and privacy violations”
Free Speech TV is a publicly-supported, independent, non-profit TV channel that is a project of Public Communicators, Inc., a 501(c)3 non-profit, tax-exempt organization. It is available nationally, 24/7, on DISH Satellite Network. Selected programs are available on 172 community access cable stations in 38 states. The Free Speech social networking community can be found at http://freespeech.org


Paper Tiger Television 25th Anniversary Celebration
Mon, 08 Jan 2007 09:32:44 +0000
MFPL member Paper Tiger Television is celebrating a full 25 years of life Thursday, October 11 at Anthology Film Archives. I was an active member of Paper Tiger for 10 years, making me partial to the organization. Nonetheless, I think I can objectively say that Paper Tiger has been a critical part of the alternative media movement for all of its 25 years and with our support will continue to be for the next 25. Hope to see you there! Full details are here.


People for Internet Responsibility
Mon, 08 Jan 2007 09:32:44 +0000
Thanks Michael for the tip on People for Internet Responsibility.
They have started an interesting <a href=“http://forums.pfir.org/>set of forums where issues like the definition of spam are being debated. Definitely worth the read.


Police Brutality Strikes Fifth Anniversary of Sylvia Rivera Law Project
Mon, 08 Jan 2007 09:32:44 +0000
Please support MFPL member SRLP!!
On the night of Wednesday, September 26, officers from the 9th Precinct of the New York Police Department attacked without provocation members of the Sylvia Rivera Law Project and of its community. Two of our community members were violently arrested, and others were pepper sprayed in the face without warning or cause.
PRESS RELEASE:
FOR IMMEDIATE RELEASE
Contacts:
Jack Aponte (jack@srlp.org, 347-247-1526)
Naomi Clark (naomi@srlp.org, 917-907-4870)
Police Brutality Strikes Fifth Anniversary of Sylvia Rivera Law Project
NEW YORK - On the night of Wednesday, September 26, officers from the
9th Precinct of the New York Police Department attacked without
provocation members of the Sylvia Rivera Law Project and of its
community. Two of our community members were violently arrested, and
others were pepper sprayed in the face without warning or cause.
The Sylvia Rivera Law Project (www.srlp.org) is an organization that
works on behalf of low-income people of color who are transgender,
gender non-conforming, or intersex, providing free legal services and
advocacy among many other initiatives. On Wednesday night, the Sylvia
Rivera Law Project was celebrating its fifth anniversary with a
celebration and fundraising event at a bar in the East Village.
A group of our community members, consisting largely of queer and
transgender people of color, witnessed two officers attempting to
detain a young Black man outside of the bar. Several of our community
members asked the officers why they were making the arrest and using
excessive force. Despite the fact that our community was on the
sidewalk, gathered peacefully and not obstructing foot traffic, the
NYPD chose to forcefully grab two people and arrested them. Without
warning, an officer then sprayed pepper spray across the group in a
wide arc, temporarily blinding many and causing vomiting and intense
pain.
“This is the sort of all-too-common police violence and overreaction
towards people of color that happens all the time,” said Dean Spade,
founder of the Sylvia Rivera Law Project. “It’s ironic that we were
celebrating the work of an organization that specifically opposes
state violence against marginalized communities, and we experienced a
police attack at our celebration.”
“We are outraged, and demand that our community members be released
and the police be held accountable for unnecessary use of excessive
force and falsely arresting people,” Spade continued.
Damaris Reyes is executive director of GOLES, an organization working
to preserve the Lower East Side. She commented, “I’m extremely
concerned and disappointed by the 9th Precinct’s response to the
situation and how it escalated into violence. This kind of aggressive
behavior doesn’t do them any good in community-police relations.”
Supporters will be gathering at 100 Centre Street tomorrow, where the
two community members will be arraigned.  The community calls for
charges to be dropped and to demand the immediate release of those
arrested.

END -



Printing and Samba
Mon, 08 Jan 2007 09:32:44 +0000
Finally working on getting the windows clients to print using a printer install on a Samba server, rather than directly via IP.
First step, install cupsys and cupsys-client:

apt-get install cupsys cupsys-client

Then install some ppd files:

apt-get install foomatic-filters-ppd

Then, I went to Open Printing to lookup the best PPD file for the printer in question.
I searched in /usr/share/ppd for that PPD, copied it to my home directory, and then installed the printer with:

lpadmin -p NameOfPrinter -E -v socket://ip.ad.dr.ess:9100 -P LaserJet_1320-hpijs.ppd

I made sure it was there with:

lpstat -v

Then I made it the default printer with:

lpadmin -d NameOfPrinter

Finally I tested by creating a test.txt file and typing:

lp test.txt

And now the Samba part:
Make sure you have the following in the global section of your /etc/samba/smb.conf file:
printing = cups
printcap name = cups
cups options = "raw"
load printers = yes

And the following two share definitions:
[printers]
comment = All Printers
path = /var/spool/samba
browseable = no
public = yes
client driver = Yes
guest ok = yes
writable = no
printable = yes
printer admin = root
;  create mode = 0700

[print$]
comment = Printer Drivers
path = /var/lib/samba/printers
browseable = yes
guest ok = no
read only = yes
write list = root

Next - fire up a windows computer (note: I can’t get this to work on Windows
Vista, but it does work on Windows XP).
Then, browse your network shares to find your samba server. Double click on the
printers folder (not on the printer that you see listed). Right click on the
printer in the printers folder and click Properties. From the advanced tab,
select to install a driver. Navigate to a folder containing the drivers (which
you should download from the driver manufacturer).
NOTE: If you are adding a new printer you seem to need to restart samba to get
it to show up (reload didn’t seem to work). I think that’s because it’s the
nmbd service that needs to restart.


Progressive Technology Project is sponsoring TechCamp in Minnesota
Mon, 08 Jan 2007 09:32:44 +0000
The Progressive Technology Project, a member of May First/People Link for years, is hosting a 3 day technology “boot camp” for organizers.  It is is    designed to build the basic skills of community organizers in the key    technology competencies for community organizing.
For more information, see their  web site.



Radical Techies: The USSF needs you!
Mon, 08 Jan 2007 09:32:44 +0000
Dear radical technologists: The social forum needs you! We’re expecting thousands of progressives to converge in Atlanta the end of June for one of the largest gatherings of social justice activists in recent history (for more info: www.ussf2007.org). At the moment, hundreds of workshops have been proposed with depressingly few covering any type of issues related to the Internet.
To wit:
mysql> SELECT count(*) FROM node WHERE type =
'content_ussf_2007_session_propos' AND title REGEXP
'(tech|computer|secure|free software|open source|internet)';

+----------+
| count(*) |
+----------+
|        4 |
+----------+
1 row in set (0.06 sec)

mysql>
And none of the four are about the Internet (one is about High Tech companies and pollution, one is about electoral organizing, and one is about media skills). One - which looks excellent, is called “Building Technology Skills; Building Movement: Organizing Technology Community of Practice” and is being organize by the Progressive Technology Project.
Despite the promising title from PTP, we’re way under represented in the program.
Fortunately, the is still over a week left to propose more sesions. At May First/People Link we brainstormed some sessions that we think are particularly important. We’re sending this out to you because we know you’re good and we think any number of you can run them. Please consider - and submit a proposal!!
Proposal Ideas:
Secure and Private email: Learn about how email works, when you should consider using encryption and when you shouldn’t, and find out how you too can share email messages without worrying about being spied on by the man.
Secure Internet Browsing: what does that lock on my browser mean? Come to this session and learn how to surf carefully, understanding how to make concious decisions as you surf about what sites you want to connect to and which sites you want to enter private data on.
Using Free Software: There is a lot of free software you can install on your comptuer today. Learn about what software is available and why it’s important to support.
To submit a proposal - go to:
www.ussf2007.org
See you in Atlanta!!


Setting up OpenId
Mon, 08 Jan 2007 09:32:44 +0000
There are less than a dozen technologies on the Internet that I would consider truly critical to organizing on the Internet, and one of them is a distributed authentication system. OpenID is the most promising implementation.
The OpenID short description is:

OpenID is a URL based identity system. An OpenID identity is simply a URL, and all the OpenID specification does is provide a way to securely prove that you own that URL.

Unlike most identity systems, OpenID is completely decentralized. The end-user may choose whichever server they are most comfortable with, and may even run their own if they choose.
Normally, everytime we create a new web site, we have to build a authentication system or use whatever authentication system comes with the development platform we are building. If I install Drupal, then we use the Drupal users table and we have to go through the process of creating usernames and passwords for people who want to login to the site.
If we build custom databases, we have to write our own system for storing usernames and passwords.
The result is that everyone has a million usernames and passwords and most of them have been communicated over insecure email.
Yuck.
With OpenID - we can create a single web-based authentication system that can be shared by any web application we create. What’s even better: by adhering to the OpenID standards, our web applications can communicate with other OpenID servers, and other OpenID web applications can communicate withour OpenID server.
So - my goal here was to setup an OpenID server for May First/People Link that used our existing database of user accounts as the user/database store. With this in place, any web application in existence now that support OpenID (like LiveJournal) will allow users to login with their May First/People Link username and password.
Here are my experiences in setting up an OpenID server that would allow May First/People Link users to login to any OpenID enabled client using their existing user account login and password.
Before beginning - a note on Open ID URLs:
Your OpenID identity is written as an URL. The exact format of the URL is up to the person who sets up the OpenID server.
For example, it is often written as:
http://username.domain.org
But it could just as easily be written as:
http://domain.org/username
Or any variation.
Regardless of how it is written, the URL must display a valid web page in which there is a header tag (invisible to the person browsing the web page) that contains:
&lt;link rel="openid.server" href="https://members.mayfirst.org/openid/server.php" /&gt;

The href part refers to the server address that will authenticate the user.
In other words, the server software to do the authentication is not part of your identity URL. Instead, your identity URL refers to a web page that contains the URL for the server software that will authenticate you.
So… I found it quite easy to create a simple web site that responds to:
http://id.mayfirst.org/any-username-you-want
But much harder to create a simple web site that responds to:
http://any-username-you-want.mayfirst.org
which influenced my decision to use the OpenID URL format of:
http://id.mayfirst.org/username
This might be a bad idea, maybe it’s a good idea. Not sure.
Anyway… here are the steps I took:


Setup up id.mayfirst.org web site (on harry.mayfirst.org, but it could really go anywhere).


Installed the Jan Rain libraries with:

pear install http://www.openidenabled.com/resources/downloads/php-openid/pear/Auth_OpenID-1.2.2.tgz



following the instructions here: http://www.openidenabled.com/openid/libraries/php
That step install the php Pear libraries in /usr/share/php.
Next I downloaded the same thing and unzipped it.
It contains sample scripts for setting up a server.
I modified the server files so that they would authenticate against our user accounts database.
Then I ran the tests here:
http://www.openidenabled.com/resources/openid-test/checkup
It was a little confusing because some tests require you to be logged in and some require you to be logged out, but once I figured that out all of the tests passed for us.
So - if you want to test it yourself - find a web site that supports openid - and try it out.
Your openid URL is:
http://id.mayfirst.org/username
Replace username with your real username and it should work.
Note: I’m not publicizing this to our members because I’d like us to test it and make sure that the URL scheme is the right one. Please give feedback!


Supporting small organizations
Mon, 08 Jan 2007 09:32:44 +0000
After doing one thing for over eight years, one would usually expect to have a certain level of competence in that activity. One would even expect to say: “I know know how to do X.”
However, when it comes to developing sustainable relationships with small organizations involving technology and the Internet, I realize that I just don’t know how to do it.
And - I don’t think I’m alone. In fact, I’ve worked and continue to work closely with dozens of very smart people, and we just can’t seem to figure it out.
Here’s the scenario:
A small group (less than 10 paid staff, often no paid staff) has a web site or online database. Perhaps the relationship starts with me building that web site or database. In either event, they periodically need changes made. No database or web site is ever finished. As long as the organization is growing and changing, so is there database and web site.
The problem is that these changes come in dribbles. Six months go by with no changes. Then, a 10 minute change is need. Or a thirty minute change is needed. Often, these changes come with a huge urgency. Something is not working that is needed immediately.
Sometimes the organization can pay for the changes and expect to, sometimes the organization can’t pay for the changes and expect them for free. Sometimes the changes reveal a bug in the first implementation. Sometimes organizations understand that bug fixes are a normal part of any technology deployment, other times organizations think the bugs are like a product defect and fixing them for free is the responsibility of the developer.
The problem is that the relationship becomes one based on exchange: the organization needs X changed. The developer will deliver X at a price of $Y.
There are a number of bigger political problems with the exchange model, which we could discuss for hours, and I think that’s a good discussion.
For now, however, I want to only address the impracticality of that model. Developers, like all humans on this planet, need a steady income in order to live. Even when the organizations are available and willing to pay for all changes, this model is a really hard one to live on. The pay is too unpredictable and too little. The only way to survive on this model is to over commit and over promise.
There are some solutions to this problem, but they all have draw backs:

Charge more money for hourly work. This only solves the “too little” problem - it doesn’t solve the random problem. And, most small organizations can’t afford to pay more
Only work with large organizations. Yes, this works. Just not for small organizations
Collectivize the work. May First Technology Collective did this for 6 years and had a lot of success. However, we eventually closed because we could not afford to support the small organizations. We either needed to be subsidized (for a while we had bartered space and relied on a lot of foundation contracts) or we had to tip the scale in favor of much larger organizations. The difference between an individual making $60/hour and an organization paying staff salaries and health insurance making $60/hr is enormous.
Scheduled support. This might be the closest to a working model. Pick a time interval - 4 hours once a month, 2 hours ever two months. Schedule the days that the time will be worked in advance. One week before the day start communicating about what changes will be made. On the day in question, do the work. This has serious drawbacks for the organization in question: often changes need to be made right away and this model doesn’t account for it. On the other hand, it does allow the developer to schedule themselves in a way that can guarantee their time.

What other models are there?


The future of May First/People Link backups
Mon, 08 Jan 2007 09:32:44 +0000
Hi folks - this is an open call to May First/People Link Members inviting feedback on our backup strategy. All thoughts and comments welcome!
About 2 - 3 weeks ago we changed our backup strategy. We had been running two offsite backup servers in our sunset park office (in addition to one backup server in our hosting facility in Deleware which only provides a redundant back up of the After Downing Street database). We needed a change for two reasons:

Both of the sunset park servers’ hard drives were nearly at capacity
Neither server was running a redundant disk array. That means that a single hard drive failure would require us to re-copying all the backup data - a process that would take either 4 - 5 weeks to do over the Internet, or a physical trip to our rack in Telehouse plus a trip to the 3 - 4 member offices that do a remote backup to our servers.

Our new strategy was to build a single backup server with multiple disks using a RAID array. That means if one disk fails, the system continues running without a problem. This single backup server, named Iz, was brought online with 8 hard disks about 2 - 3 weeks ago.
We immediately ran into overheating problems, which caused the entire server to emit a continuous ear piercing beep from the depths of hell.
We solved this problem by opening the case and fixing a couple broken fans.
This worked for a couple weeks. Then, last friday, one disk failed. Since it is a redundant system, our backups continued running smoothly. On Tuesday, Daniel and I spent the entire day replacing the faulty disk. It should have taken 30 minutes, but we were slowed down by the fact that our RAID setup is very complicated, our bootloader was confused about which of the 8 disks actually contained our kernel, and, most troubling, we discovered hardware failures related to the hard drive controllers - leaving us still unsure if it was actually a disk failure or a controller failure. By the end of the day, Iz was back in place, with the disk replaced (plugged into a different controller). The particular disk that failed was acting as a spare on the giant RAID array, so the replacement disk was nearly immediately incorporated in the RAID array and everything went back to normal.
The very next morning another disk failed. This time the failed disk was not a spare. Iz continued functioning properly (as it should) and additionally it incorporated the spare into the RAID array. Incorporating a new disk into a RAID means “syncing” the data from the functioning RAID array to the newly incorporated disk. This process went on all day Wednesday and through the night, causing several of our normal backups to be delayed since the normal backups and the syncing process were both competing for resources on the server. On Thursday morning (yesterday) I noticed that several of our backup scripts were still running and the new disk was still syncing.
At this point we suspected that heat was the cause of the hardware failures and decided that the best course of action would be to spread out the hard drives inside the box (they were all placed tightly next to each other in two cages during our original installation).
However, in what order do we proceed? Until the syncing completes, one more hard drive failure will cause us to lose all the data on the entire server (yes, it is backup data, but re-copying that data is no small feat). If we let the backups run their course, the syncing will take longer. The longer we wait to spread out the hard drives, the higher the odds that another drive will fail.
In the end, I stopped the still running backups to allow the syncing to occupy all the system resources. By 8:00 pm Thursday night, it still was only 85% done, reporting that it need another 10 hours. Nervous that another hard drive might fail, I shutdown Iz and spread out the hard drives. When I re-booted, the RAID array didn’t come back properly. On Alfredo’s excellent advice, I went home rather than push my luck trying to fix this by myself after a 12 hour work day.
This morning (Friday), with a much clearer head, I discovered the drive that didn’t come up properly, reseated the cables, and now Iz is back and running.  It’s still syncing, but it appears that it should complete before the backups run again tonight, meaning we should be back in shape.
For now.
The whole point of this long post is to propose a new way of doing backups.  Alfredo was really pushing Daniel and I early this week to re-consider Iz - I wasn’t ready to listen to that before, but now I’m seeing the wisdom in his concerns, so here are some ideas:
New problems we have with our new backup strategy:

Iz has hard ware problems (particularly related to the hard drive controllers). It’s very possible that all hardware problems are related to heat generated by eight hard drives, and figuring out a way to cool down Iz will solve all problems. However, it might be that Iz has faulty hardware, or that the heating we’ve already subjected Iz to has caused permanent damage. In any event, I don’t have a lot of confidence in Iz.
We can withstand one disk failure. That’s a huge improvement. However, two disk failures will cause us to lose all data because we now only have one backup server for all our data. Related: if Iz has a motherboard failure, we will be offline until we can find a replacement server that can handle 8 disks. And - all backup activity will stop until do that.
Our only backup is offsite. That’s a good place for a single backup (much better than onsite!).  However - if one of our servers fails at Telehouse - we will have to either copy the replacement data over the Internet (which will take hours if not days), or physically carry Iz to Telehouse and re-copy. This is not such a great strategy.

Proposed new strategy:
We have about 450 GB of backup data currently. About 375 GB is from our Telehouse server and the remaining 75 GB is from member office backups.

We phase out Iz in favor of multiple backup servers with 3 300 GB RAID 5 disk arrays. This will give us 600 GB per backup server with one redundant hard drive. It will spread out our backups over multiple machines.
We place backup servers both at Telehouse and Sunset Park. All of our Telehouse production servers will backup to both the Telehouse and Sunset Park servers. All of our member backups will only go to Sunset Park (at Telehouse we’re charged for bandwidth, so backing up members to Telehouse is not economically feasible).

Proposed Steps:

Install a single new backup server in Telehouse and start backing up our production servers. We’ll be using 375 of the 600 available GB, so we should be in good shape for now.
As a temporary measure, make a single backup of all member backup data currently on Iz to Richie. Richie is one of our old backup servers. If we delete all the May First Telehouse servers data from Richie (which we can safely do once we have have the Telehouse backup server in place), there will be plenty of room on Richie. This step is designed to allow us to recover if we have two hard drive failures on Iz, which would result in losing member backup data.
Install a single backup server in Sunset Park to take over for Iz.

Now, we can retire Richie and Retire Iz and re-use their hard drives for building our more backup servers.


The ongoing project of understanding load and bottle necks
Mon, 08 Jan 2007 09:32:44 +0000
I’ve found bottle neck analysis to be something I can only understand in small bite sized chunks - really it’s a continuous process of peeling back layers.
Thanks to yet another conversation with dkg, I’ve peeled back more layers.
Some general ideas to keep in mind:
The load average as reported by top is essentially the number of processes  that are trying to run. As Wikipedia puts it:

An idle computer has a load number of 0 and each process that is using CPU or waiting for CPU adds to the load number by 1.
It’s a good indicator of a sluggish machine, but doesn’t tell us why processes can’t run. Two common reasons could  be that the CPU is overtaxed or processes need to write or read from the disk (commonly referred to as disk i/o for disk input/output), but there is so much reading and writing to the disk that they are waiting around for the kernel to give them the info they want or to write the data they want.
So, in comes vmstat (typically run with vmstat 1 to show a report every one     second - it does not require root to run). The first two columns are the most important. The first, “r”, is “The number of processes waiting for run time.” The second is “The number of processes in uninterruptible sleep” (or blocked), which I guess technically doesn’t always mean due to i/o, but seems like that’s the most likely cause. (The quotes come from the vmstat man page).
If the number in the “r” column is high, but the number in the “b” column is low, then the contention is probably over the CPU. If the number in the “b” column is high, then most likely it’s a disk i/o problem.
If you have a large number in the “b” column, you’ll want to pay attention to the si and so columns under SWAP. If those numbers are over 0 then the problem is probably swap, meaning that you don’t have enough RAM in the machine so the kernel is using the hard drive as a RAM storage space to compensate.
If the si and so numbers are zero, then the problem may be processes that are hard disk intensive.
In this case, running ps and greping for the state “D” (for example: ps aux | grep " D “) will produce a list of the processes in question that are blocked. Running lsof and grepping for those processes could tell you what files are being opened by them which might give some clues as to which process is the culprit. NOTE: if you are experiencing high load running lsof will only contribute to your load problems!
One more point of confusion: You might wonder how there could be, say, 10 processes in the “b” column. But, on the far right, 0% of the CPU is “wa” (meaning waiting, usually on i/o). That can happen if a bunch of processes are waiting on i/o and the kernel says: well, instead of sitting around idle waiting on i/o, I’m going to take care of the processes that do not require i/o until the i/o bottle neck gets cleared up. In that scenario, the cpu will have 0% of it’s time waiting even though a number of processes are blocked because of i/o problems.
Hope this is helpful for others. Thanks dkg and man and wikipedia!


This is how hot I am
Mon, 08 Jan 2007 09:32:44 +0000
Now that Chavez can accurately report the temperature (see previous blog) I needed to know what was going on when the kernel reported a temperature spike. The kernel had been reporting, every several weeks messages along these lines:
Nov 17 08:02:39 chavez kernel: CPU0: Temperature above threshold
Nov 17 08:02:39 chavez kernel: CPU0: Running in modulated clock mode
So, I installed Swatch, a tool that monitors log files and takes action depending on what it finds.
I created the following .swatchrc file and put it in the home directory of a user in the adm group (which allows them to read the log files):
watchfor /chavez kernel: CPU/
threshold on
threshold chavez,type=limit,count=1,seconds=60
# $_ gets replaced with the syslog line that matches
exec "echo $_ >> /home/mayfirst/swatch.log"
exec "top -b -n 1 | head  -20  >> /home/mayfirst/swatch.log"
exec "sensors | grep temp >> /home/mayfirst/swatch.log"

Then, I ran it with:
swatch --tail-file /var/log/syslog --daemon

I love swatch with one major exception - I can’t get it to properly restart if the log file has been rotated out from under it (it needs a deadly kill -9 to end the script).
Fortunately, it only took two days to get a temperature spike. The spike happened while upgrading Drupal from 4.6 to 4.7. The upgrade process drops and adds fields in tables that have massive amounts of records in them (or at least the drupal site I was upgrading did). So - the intense mysql activity seems to have caused the spike (hard drive activity or processor activity - I still am not sure).
The normal sensors temperature is about 40 - 50 C. The kernel alarm went off when the temperature reached 70 C. Running in modulated clock mode seemed to be very effective - as the temperature never got above 72 C.


Two opportunities for US Social Forum Report back in NYC
Mon, 08 Jan 2007 09:32:44 +0000
The The Foundry Theatre/SOLAS Saturday, July 21st, 2007 5-9PM and the Brecht Forum: Wednesday, July 25, 7:00 pm
The Brecht Forum invites all who participated in the first US Social Forum in Atlanta, June 27-July 1, and those who would like to hear about it, to come together for an open discussion of what happened and what was accomplished in this major effort to help develop the leadership, consciousness, vision, and strategy needed to realize another world.
The Brecht Forum
http://brechtforum.org
July 25, 7:00 pm
451 West Street (that’s the West Side Highway) between Bank & Bethune Streets
The Foundry Theatre is hosting our version of a report-back on Saturday, July 21st. We’ve invited all of our friends and colleagues from the New York City based downtown arts community who were unable to travel to Atlanta, and we’d love it if all of you who did make it could join us.
Less report back than bar-night, we’ve decided to reserve the 2nd Floor of one of our favorite local haunts from 5-9PM. We’ll be there all evening, you could come and go as you please, stop by to say hello and re-connect, or stay all evening along with. We hope you’ll join us and share your experience and your work with some wonderful people from our community (over a beer or two).
The Foundry Theatre / USSF Bar Night
http://thefoundrytheatre.org
Saturday, July 21st, 2007 5-9PM
SOLAS Bar
232 E. 9th St., New York, NY 10003
between Second and Third Aves.


Upgrading assata from sarge to etch
Mon, 08 Jan 2007 09:32:44 +0000
This server is primarily a mailman server. That’s what it does and, since we’re moving the lists it is hosting to a new server, it won’t do that for much longer.
The target server will be running etch, while Assata is currently running sarge. So, I wanted to upgrade Assata to etch first (or at least upgrade mailman to etch).
I started by editing /etc/apt/sources.list changing references from sarge to etch.
Then I ran:

apt-get update
apt-get install mailman

I was hoping to get off easy. No luck.
root@assata:~# apt-get install mailman
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
busybox coreutils klibc-utils libc6 libc6-dev libklibc libselinux1 libsepol1
libvolume-id0 locales lsb-base makedev module-init-tools python-support
tzdata
Suggested packages:
glibc-doc manpages-dev python-korean-codecs python-japanese-codecs listadmin
python2.3-korean-codecs python2.2-korean-codecs
The following packages will be REMOVED:
base-config initrd-tools kernel-image-2.6-686 kernel-image-2.6.8-2-386
kernel-image-2.6.8-3-686
The following NEW packages will be installed:
busybox klibc-utils libklibc libselinux1 libsepol1 libvolume-id0
python-support tzdata
The following packages will be upgraded:
coreutils libc6 libc6-dev locales lsb-base mailman makedev module-init-tools
8 upgraded, 8 newly installed, 5 to remove and 248 not upgraded.
Need to get 24.4MB of archives.
After unpacking 71.7MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.
root@assata:~#

I really don’t want to remove the kernel. Remembering my experiences from previous upgrade Itried instead:

root@assata:~# apt-get install initrd-tools

And then tried:

apt-get install mailman

This time I got:
root@assata:~# apt-get install mailman
Reading Package Lists... Done
Building Dependency Tree... Done
The following extra packages will be installed:
lsb-base python-support
Suggested packages:
python-korean-codecs python-japanese-codecs listadmin
python2.3-korean-codecs python2.2-korean-codecs
The following NEW packages will be installed:
python-support
The following packages will be upgraded:
lsb-base mailman
2 upgraded, 1 newly installed, 0 to remove and 252 not upgraded.
Need to get 8648kB of archives.
After unpacking 8978kB of additional disk space will be used.
Do you want to continue? [Y/n]

That’s better.


Upgrading Sarge -> Etch
Mon, 08 Jan 2007 09:32:44 +0000
After updating my /etc/apt/source.list, the first step was to upgrade the kernel image. I tried:
sudo apt-get install linux-image-2.6-686

But got unmet dependencies issues. So I tried:
sudo apt-get install linux-image-2.6.18-3-686

I was in way too much of a hurry and didn’t notice that apt decided to uninstall grub. Not sure why Got the following:
Setting up linux-image-2.6.18-3-686 (2.6.18-7) ...
Running depmod.
Finding valid ramdisk creators.
Using mkinitramfs-kpkg to build the ramdisk.
readlink: invalid option -- m
Try `readlink --help' for more information.
readlink: invalid option -- m
Try `readlink --help' for more information.
initrd.img() points to  (/boot/initrd.img-2.6.8-3-686) -- doing nothing at /var/lib/dpkg/info/linux-image-2.6.18-3-686.postinst line 585.
readlink: invalid option -- m
Try `readlink --help' for more information.
readlink: invalid option -- m
Try `readlink --help' for more information.
vmlinuz() points to  (/boot/vmlinuz-2.6.8-3-686) -- doing nothing at /var/lib/dpkg/info/linux-image-2.6.18-3-686.postinst line 585.
The provided postinst hook script [/sbin/update-grub] could not be run.
dpkg: error processing linux-image-2.6.18-3-686 (--configure):
subprocess post-installation script returned error exit status 2
dpkg: dependency problems prevent configuration of linux-image-2.6-686:
linux-image-2.6-686 depends on linux-image-2.6.18-3-686; however:
Package linux-image-2.6.18-3-686 is not configured yet.
dpkg: error processing linux-image-2.6-686 (--configure):
dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of kernel-image-2.6-686:
kernel-image-2.6-686 depends on linux-image-2.6-686; however:
Package linux-image-2.6-686 is not configured yet.
dpkg: error processing kernel-image-2.6-686 (--configure):
dependency problems - leaving unconfigured
Errors were encountered while processing:
linux-image-2.6.18-3-686
linux-image-2.6-686
kernel-image-2.6-686
E: Sub-process /usr/bin/dpkg returned an error code (1)

So I tried:
sudo apt-get install grub

I got chastized with:
You shouldn't call /sbin/update-grub. Please call /usr/sbin/update-grub instead!

But it worked. Now we’re getting some where. Next I did a reboot.
All came up fine.
And then, because it is primarily a samba machine:
sudo apt-get install samba

And finally:
sudo apt-get dist-upgrade



Upgrading to lenny
Mon, 08 Jan 2007 09:32:44 +0000
In a rather non-thinking moment, I decided to upgrade Liberace, my laptop machine, to Lenny when I really didn’t have time to recover from problems. Remarkably, I only encountered one which took 5 minutes to deal with!
The Etch release of Debian became the “stable” release yesterday, meaning liberace, which is always tagged to get the “testing” release has now moved on to “Lenny” - the new “testing” release.
After about 45 minutes of downloading and installing (thanks apt-get dist-upgrade) I was ready to go. I only ran into one problem, with acpid:

Starting Advanced Configuration and Power Interface daemon...acpid: can't open /proc/acpi/event: Device or resource busy

I tried:

sudo /etc/init.d/acpid stop
sudo /etc/init.d/acpid start

But got the same error. Then I tried:

0 jamie@liberace:$ sudo lsof | grep /proc/acpi/event
acpid      4529     root    3r      REG        0,3        0 4026532119 /proc/acpi/event
0 jamie@liberace:$ ps aux | grep 4529
root      4529  0.0  0.1   1580   632 ?        Ss   Mar31   0:00 /usr/sbin/acpid -c /etc/acpi/events -s /var/run/acpid.socket
jamie     1991  0.0  0.1   1956   628 pts/10   R+   12:56   0:00 grep 4529
0 jamie@liberace:$

Ok, so it really is acpid that is holding on to this file. Then:

0 jamie@liberace:$ sudo kill 4529
0 jamie@liberace:$
0 jamie@liberace:$ ps aux | grep 4529
jamie     2005  0.0  0.1   1956   632 pts/10   R+   12:56   0:00 grep 4529
0 jamie@liberace:$ sudo /etc/init.d/acpid start

And now we’re back in business. Seems like acpid was not shutting down properly.


Useful trick for changing resolutions
Mon, 08 Jan 2007 09:32:44 +0000
I was helping a May First/People Link member with a projector presentation (my personal laptop, liberace, was drafted into service) yesterday when I realized that liberace’s screen resolution was too wide for the projector to handle.
I’m familiar with the solution on a desktop computer (that has the numbers keypad): Ctrl-Alt-Keypad+ and Keypad- to dynamically change your resolution. However, liberace, like most laptops, doesn’t have a keypad. After futzing with the projector, I finally got it to display the screen properly, however, I really could of used a method for changing the resolution.
After a little bit of research I discovered the xrandr command. When run with no arguments, it displays your available resolutions:
0 jamie@liberace:~$ xrandr
Screen 0: minimum 320 x 200, current 1280 x 800, maximum 1280 x 1280
VGA disconnected (normal left inverted right x axis y axis)
LVDS connected 1280x800+0+0 (normal left inverted right x axis y axis) 261mm x 163mm
1280x800       65.3 +   60.0*
1280x768       60.0
1024x768       60.0
800x600        60.3
640x480        59.9
TMDS disconnected (normal left inverted right x axis y axis)
0 jamie@liberace:~$

You can switch resolutions with:
0 jamie@liberace:~$ xrandr -s 800x600 -r 60
0 jamie@liberace:~$

You don’t even need to be root!


Using Goddy SSL certificates
Mon, 08 Jan 2007 09:32:44 +0000
After some agonizing discussions, dkg and I decided to purchase a SSL certificate from Godaddy for members.mayfirst.org and id.mayfirst.org. It was agonizing - when our Dotster/Thawte purchased SSL certificate for members.mayfirst.org expired a month ago, I tried to purchase another one from Dotster/Thawte, but dotster’s tech just is not up to snuff. In the last year, half the time I purchase from them it works, and the other half the time I just never get the certificate. Contacting support doesn’t help (they say it’s Thawte’s problem, the clearly more competent Thawte people say it’s Dotster’s fault). In these interactions I’ve tended to believe Thawte.
All of this is even more frustrating given what a scam SSL certificates are (see dkg’s piece on that topic for an in-depth discussion.
We scoured the Internet for affordable SSL certificate issuers and discovered the godaddy is really the cheapest. Of course, we hate godaddy. However, we decided we needed a cert that will not cause our members to get in the habit of seeing ssl error messages and ignoring them.
Imagine my frustration when, after installing godaddy’s cert, my Firefox browser still gave the error!
Fortunately - with some help on the Internet, I discovered that the answer to the problem is:
When you get the email from Godaddy about your cert being ready for download, click on the link and you get a zipped package with both your certificate and a “godaddy intermediate bundle.” I didn’t click on the link, I simply logged into the web interface and downloaded my new certificate. The intermediate bundle is a file containing three certificates - essentially a chain of certificates that connects the godaddy certificate authority to whatever certificate authorities your browser trusts.
If for some reason you don’t have this bundle, it’s attached to this post below.
Copy the file to our server and then, in your Apache file, add this line:
SSLCertificateChainFile /etc/apache2/ssl/gd_intermediate_bundle.crt

Then restart Apache.
Useful resources:
https://certificates.godaddy.com/InstallationInstructions.go
http://www.dslreports.com/forum/r18609704-New-SSL-Cert-Problem-Report~start=20


Virtual Folders in Mutt
Mon, 08 Jan 2007 09:32:44 +0000
Just found a great how to on implementing virtual mail folders in Mutt. The problem: I want to be able to search for a message and have it find messages across multiple mailboxes.
The fix came from a really useful site that I had to use google cache to view. Sigh. Maybe they want to remain anonymous. Not sure. In any event, here’s the info:
First install mairix:
apt-get install mairix

Next download mailplex (a python script). I found it here. It’s attached to this blog if it disappears. Don’t forget to chmod 775 it (and you can remove the .txt extension - I only did that to outsmart Drupal).
Next: create a mailbox called .results:
cd ~/Maildir
maildirmake .results

Now, edit your .muttrc file:
~/.muttrc
# Mairix bindings
macro browser z "&lt;shell-escape&gt;mailplex\n"
macro index z "&lt;shell-escape&gt;mailplex\n&lt;change-folder&gt;=.results\n"
macro pager  z "&lt;shell-escape&gt;mailplex\n&lt;change-folder&gt;=.results\n"

Create a mailplex config file
~/.mailplex
# mailplex config (uses mairix)

mairix_confs = [
'~/.config/mutt/mairix-conf'
]
output_dir = '~/Maildir/.results'

Create a mairix-mutt config file:
~/.config/mutt/mairix-conf
# mairix config file for jamie's archived mail

base=/home/jamie/Maildir

# recursively scan Maildirs
maildir=.:.Trash:.saved:.Sent

And finally create a cron job:
30 * * * * mairix -f $HOME/.config/mutt/mairix-conf



War & the Media: Book Party and Forum at the Brecht Forum
Mon, 08 Jan 2007 09:32:44 +0000
Support your fellow May First/People Link member the Brecht Forum this Wednesday March 28 at 7:30 pm by attending their forum about War and the Media with superstars Jeff Cohen and Robin Andersen. Looks like a great event for both getting information and participating in a discussion about how the war is being mis-covered in the media. Please attend!
Brecht Forum: 451 West Street, (212) 242-4201
Co-sponsor: FAIR
War & the Media
The Battle for Public Perception
Robin Andersen & Jeff Cohen
Media critics and analysts Andersen and Cohen will discuss the connections between the media and war and the challenges this presents for the left. They will explore how the selection of which stories are told and which are ignored helps justify past battles and ensure future wars as well as the ways new technologies have helped turn the nightly news and talk show into “infotainment.”
Robin Andersen is director of the Peace and Justice Studies Program at Fordham University and author of A Century of Media, A Century of War. Jeff Cohen is the founder of Fairness and Accuracy in Reporting (FAIR) and author of Cable News Confidential: My Misadventures in Corporate Media, a memoir of his experiences as a pundit with FOX News, CNN and MSNBC.
Suggested donation: $6/$10/$15
No one turned Away
451 West Street, New York, NY 10014
(212) 242-4201
email: brechtforum at brechtforum.org


Where's May First this month?
Mon, 08 Jan 2007 09:32:44 +0000
Like most of you, we’re in Atlanta this month for the US Social Forum!
And, following the lead of our members, we have made a tremendous commitment to the supporting the social forum, starting back in October when we began actively organizing the Information and Communications Technology committee (ICT). The ICT comittee is heavily represented by May First/People Link staff and members, including folks from member organizations Interactivist and Openflows. And we are donating all the bandwidth and a dedicated server to the project.
This month the work load is skyrocketing as we are making the final push to ensure a successful forum. We hope you will all be patient if we are not as quick to respond to support as we normally are. Of course, those of you in Atlanta will know where to find us :).
See you in Atlanta!


Working with RAID
Mon, 08 Jan 2007 09:32:44 +0000
We’ve been working with a giant backup server with 8 disks in a complex series of RAID arrays.
Last week, we had two disks fail from the same 7 Disk RAID 5 array within 24 hours of each other, causing the RAID to fail (and the whole server to stop responding).
When we re-booted the computer, the RAID in question was reported by /proc/mdstat to be inactive, with two disks missing:
md2 : inactive hde[0] hdk3[6] hdi1[5] hda1[4] hdm[1]
1367569600 blocks
It was missing sda3 and hdc1
Those disks had other partitions on the system that were working fine. And, we did a few read tests on the partitions in questions, and they seemed to fine. Hm.
We ran:
mdadm --examine /dev/sda3

The super block on sda3 reported that it was healthy, but that hdc1 was faulty and removed.
The same test on hdc1 reported that it and all disks were healthy.
So, it appears as though hdc1 went down first, followed by sda3.
We began the recovery with the –re-add command:
mdadm /dev/md2 --re-add /dev/hdc1
mdadm /dev/md2 --re-add /dev/sda3

Then, we tried to bring it up again:
0 root@iz:~# mdadm --assemble /dev/md2 UUID=15a4aefd:d0a95db7:934e8ae1:fce89514
mdadm: device /dev/md2 already active - cannot assemble it
1 root@iz:~#

Woops. Stop the array first:
0 root@iz:~# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
0 root@iz:~#

Then, try again:
root@iz:~# mdadm --assemble /dev/md2 --uuid=15a4aefd:d0a95db7:934e8ae1:fce89514
mdadm: /dev/md2 assembled from 5 drives - not enough to start the array.
1 root@iz:~#

Still not working. Try again with force:
0 root@iz:~# mdadm --assemble /dev/md2 --force \
--uuid=15a4aefd:d0a95db7:934e8ae1:fce89514
mdadm: forcing event count in /dev/sda3(2) from 36300 upto 36308
mdadm: clearing FAULTY flag for device 0 in /dev/md2 for /dev/sda3
mdadm: /dev/md2 has been started with 6 drives (out of 7).
0 root@iz:~#

Bingo. Not sure why it only took one of the disks, but we chose to copy the data off of it in a hurry and worry about that later.


Xen quick start
Mon, 08 Jan 2007 09:32:44 +0000
Here are the quick steps for turning a stock Debian Etch machine into a xen machine:


Install with:

apt-get install xen-hypervisor-3.0.3-1-i386 xen-linux-system-2.6.18-4-xen-686 libc6-xen



Edit /boot/grub/menu.1st. Modify the following line so that the dom0 doesn’t take up all the available RAM. This mod only gives dom0 128 MB
Xen hypervisor options to use with the default Xen boot option
xenhopt=dom0_mem=131072


Update grub and reboot

update-grub
shutdown -r now

Modify /etc/xen/xend-config.sxp. We made the following diff:
0 sontag:/etc/xen# diff -u xend-config.sxp /root/xend-config.sxp
--- xend-config.sxp     2006-11-13 09:13:37.000000000 -0500
+++ /root/xend-config.sxp       2007-06-01 08:30:20.000000000 -0400
@@ -68,7 +68,7 @@
#
# use
#
-# (network-script network-bridge)
+(network-script network-bridge)
#
# Your default ethernet device is used as the outgoing interface, by default.
# To use a different one (e.g. eth1) use
@@ -84,7 +84,7 @@
# two fake interfaces per guest domain.  To do things like this, write
# yourself a wrapper script, and call network-bridge from it, as appropriate.
#
-(network-script network-dummy)
+# (network-script network-dummy)

# The script used to control virtual interfaces.  This can be overridden on a
# per-vif basis when creating a domain or a configuring a new vif.  The
1 sontag:/etc/xen#

Install bridge-utils (for needed for networking) and xen-tools (to make creating and manipulating xen instances a lot easier)

aptitude install bridge-utils xen-tools
/etc/init.d/xend restart

Edit /etc/xen-tools/xen-tools.conf. For our purpose we made the following changes:
0 gramsci:/etc/xen-tools# diff -u xen-tools.conf.orig xen-tools.conf
--- xen-tools.conf.orig 2007-05-27 19:59:38.000000000 -0400
+++ xen-tools.conf      2007-05-27 20:01:15.000000000 -0400
@@ -35,6 +35,7 @@
#
##
# lvm = skx-vg
+lvm = vg_gramsci0


#
@@ -61,7 +62,7 @@
##
#
# copy = /path/to/pristine/image
-# debootstrap = 1
+debootstrap = 1
# rpmstrap = 1
# tar = /path/to/img.tar
#
@@ -95,7 +96,7 @@
swap   = 128Mb    # Swap size
# noswap = 1      # Don't use swap at all for the new system.
fs     = ext3     # use the EXT3 filesystem for the disk image.
-dist   = sarge    # Default distribution to install.
+dist   = etch     # Default distribution to install.
image  = sparse   # Specify sparse vs. full disk images.

#
@@ -154,8 +155,8 @@
#
# Default kernel and ramdisk to use for the virtual servers
#
-kernel = /boot/vmlinuz-2.6.16-2-xen-686
-initrd = /boot/initrd.img-2.6.16-2-xen-686
+kernel = /boot/vmlinuz-2.6.18-4-xen-686
+initrd = /boot/initrd.img-2.6.18-4-xen-686

#
#  The architecture to use when using debootstrap or rpmstrap.
1 gramsci:/etc/xen-tools#

Create new xen instances with:

xen-create-image --size=5Gb --memory=1GB --swap=512Mb --gateway=209.51.180.17 --netmask=255.255.255.240 --ip=209.51.180.24 --hostname=mendes

Change settings as needed.
Start with:

xm create -c mendes.cfg

Optionally, add more partitions:

lvcreate --size=5GB --name=mendes-var vg_gramsci0

Then, edit /etc/xen/mendes.cfg and modify the line that starts with "Disks."
Log into the console (root, no password):

xm console mendes



Sun, 08 Jan 2006 09:32:44 +0000
I saw this link on the Nonprofit Open Source Initiative listserv. I guess technology providers can’t be “neutral,” huh?
From the Reporters without Borders web site:
Another cyberdissident imprisoned because of data provided by Yahoo
Good article for learning more about corporate ISPs.
As a digression … I have to ask: what about “ordinary criminals”? Given the political nature of justice systems in China, the United States, and else where - not sure if distinguishing between political and “ordinary” criminals is useful for us.


Google's Wi-Fi Privacy Ploy
Sun, 08 Jan 2006 09:32:44 +0000
There’s lots of talk about free and low-cost wireless internet access. A city in Florida made it happen for its 30,000 people, Philadelphia is working on it and San Francisco is accepting proposals for the creation of a wireless network. Of course partnering with commercial entities comes with a price. Jeff Chester explores the potential “cost” of Google’s “free” wireless netowrk proposal in this article from the nation.
Google’s Wi-Fi Privacy Ploy
by JEFF CHESTER
[posted online on March 24, 2006]
The digital gold rush is on across America, as cities scramble to develop free or low-cost Wi-Fi zones. These public on-ramps to the Internet are designed to provide every citizen with a form of always-on, high-speed Internet access–at the playground, in the office or at home–at low or no cost.
Dozens of communities large and small, in red states and blue, are either planning or currently constructing Wi-Fi systems. Community leaders–from Philadelphia; Houston; Columbia, South Carolina; and San Francisco, to name a few–recognize that creating a citywide Wi-Fi zone is not only vital for economic development and public safety but helps insure that Americans who can’t now afford digital communications on their own can also tap in to the riches and convenience of the Internet. But there is no such thing as a free digital lunch.
Consumers and public officials should have no illusions that what is being touted as a public benefit is also designed to spur the growth of a mobile marketing ecosystem, an emerging field of electronic commerce that is expected to generate huge revenues for Google, Microsoft, AT&T and many others. Soon, wherever we wander, a ubiquitous online environment will follow us with ads and information dovetailed to our interests and our geographic location.
Unless municipal leaders object, citizens and visitors will be subjected to intensive data-mining of their web searches, e-mail messages and other online activities are tracked, profiled and targeted. The inevitable consequences are an erosion of online privacy, potential new threats of surveillance by law enforcement agencies and private parties, and the growing commercialization of culture.
Mining Your Data
Consider the application submitted to the City of San Francisco in February by search giant Google and its partner, the Internet service provider Earthlink. One of six Wi-Fi bids being considered by the City of San Francisco, the Google/Earthlink plan has attracted the most attention. Under this proposal, Google would provide a free but relatively low-speed Internet service available throughout the city (Earthlink would operate a higher-speed service on the same system charging users $20 a month). The costs of operating the “free” service would be offset by Google’s plans to use the network to promote its interactive advertising services.
Everyone who uses the Google network would first be directed to a portal page, where they would be offered an array of what Google terms “personalized consumer products.” Through those products and other technologies, Google plans, according to its proposal, to “target advertisements to specific geographical locations and to user interests.”
What this means is that Google and Earthlink plan to use online files (known as cookies) and other data-collection techniques to profile users and deliver precise, personalized advertising as they surf the Internet. (Earthlink is working with the interactive ad company DoubleClick, which collects and analyzes enormous amounts of information online to engage in individual interactive ad targeting.)
Not everyone is enthused by the Google/Earthlink model. San Francisco was advised by a trio of privacy advocates to develop policies that would respect personal privacy. In letters to the city, the ACLU of Northern California, the Electronic Frontier Foundation and the Electronic Privacy Information Center (EPIC) urged the adoption of a “gold standard” for data privacy, insuring that its Wi-Fi system would “accommodate the individual’s right to communicate anonymously and pseudonymously.” The groups also suggested that the city require any Wi-Fi company to allow users to “opt in” to any data-collection scheme. [Full disclosure: I rent office space in Washington, DC, from EPIC].
Scary Syllables
These two syllables–“opt in”–strike terror in the hearts of Google, Microsoft, AOL and everyone else in the interactive marketing field. Opting in requires users to affirmatively give permission before any data can be collected. Individuals would be fully informed about how such information would be used (such as profiling, sharing with others, etc.). What companies want instead is an “opt-out” approach, in which the default is always set to collect and make full use of our personal information.
As EPIC’s West Coast senior counsel Chris Hoofnagle explained, “The Google plan proposes to bargain away users’ privacy for a trickle of Internet connectivity.” Google will have an unprecedented ability to monitor use and build records of web activity. These records will be a honey pot for law enforcement. Individuals’ privacy is worth more than a 300K download speed." (Other Wi-Fi applicants in San Francisco also favor opt-out data-collection technology. One applicant, the NextWLAN Corporation, envisions “an e-commerce monetized, fully captive, location-aware Internet portal.” But also on the table is a proposal from the nonprofit Seakay that offers a free service and pledges no personal information will be collected online.
The interest San Francisco and other cities have in securing the financial support of commercial investors for their Wi-Fi grids in part reflects the success of the campaign run by the nation’s largest cable and phone companies, which have opposed the idea of municipally owned and operated Internet service. Companies such as Comcast and AT&T view these low-cost local municipal competitors as a threat to what they believe is their rightful broadband monopoly businesses. Already, there have been lawsuits, lobbying and legislation against such municipal Internet services.
As a result of this pressure, cities are now seeking a more corporate-friendly approach to provide what should really be a public utility operated for everyone’s benefit. Too many local governments are embracing a model for Wi-Fi, says advocate and expert Sascha Meinrath, that creates a system more favorable to “billable moments” than one designed to truly connect communities together.
Instead of creating yet another e-commerce stomping ground, San Francisco and other cities should understand that real alternatives do exist to the corporate model of municipal Wi-Fi being peddled by Google and its cohorts. It is possible to develop community networks that reflect our highest principles, including the right to personal privacy, and the cost of building such networks can be very low. There are already successful publicly supported models. St. Cloud, Florida, a city of 30,000, has built a free Wi-Fi service for its residents, seeing it as an important public service. The city has been able to build and operate the network, reduce its telecommunications costs and generate new economic opportunities.
Building a Wi-Fi network this way brings in economic development and saves the city money on telecommunications. At a time of growing media consolidation and emerging threats to the future of the Internet, America needs to create online systems that are democratically run and commerce-neutral, that protect the privacy of the citizens they serve.


A country serious about open source
Sun, 08 Jan 2006 09:32:44 +0000
Venezuela is starting an ambitious training program today. The goal is to train 400,000 people in open source software. The 24 hour courses will be free and will involve over 800 instructors. Read the press release.


Anarchist techies world wide
Sun, 08 Jan 2006 09:32:44 +0000
For the techies: a new web site for anarchist geeks has started: aptgetanarchy.org. Looks like a good beginning. My only complaint: with a name like theirs, how could they leave out “Debian” in the category list of Linux distributions!! Wait a sec, I have the ability to add new terms. Ok, never mind :).


Azad Design - accepting submissions
Sun, 08 Jan 2006 09:32:44 +0000
Check out former May First Technology Collective member Laimah Osman’s latest project: Azad Design.
From the site:

Art is social expression. Azad Design is an attempt to showcase, share and archive independent political art works.
There’s a call for submissions as well as a growing image library and other great links and resources.


Bad news in Congress for free and open communication
Sun, 08 Jan 2006 09:32:44 +0000
The Barton/COPE Act looks like bad news. Saveaccess.net highlights some of the worst aspects of the law, including weak provisions for community access television, little defense against redlining, and no defense against net neutrality.


Beyond Marriage
Sun, 08 Jan 2006 09:32:44 +0000
Anyone else feel stifled by the gay marriage debate? Really brings back unpleasant memories of the gays in the military debate. Both debates are brilliant strategy moves by the right: by defining the issue in stark black and white terms it really forces the left into a single issue position. You really can’t go full force for gays in the military AND organize against military intervention any more than you can go full force in favor of gay marriage AND question the institution of marriage in our society.
Fortunately there are smart organizers around - such as the ones that put together a statement called Beyond Marriage. Have a look at: beyondmarriae.org.


Bouncing Spam Considered Harmful
Sun, 08 Jan 2006 09:32:44 +0000
Mail is just about the hardest service to administer. Compared to web sites, databases, firewalls – just about any other service – mail is more complicated and more critical. Except the few people who use the mail servers they authored, I think few of us actually understanding all the intricacies of the settings we’re using. All of us, however, have experienced the user response when one thing gets messed up.
Given this reality, it is not surprising that mail administrators rarely want to change their mail settings. If it is working, great. Please don’t ask me to tweak it unless that tweak is really important. Even adding one small change can throw things into an incomprehensible mess of bouncing mail and upset users.
So, this is not a rant. Instead it’s a thoughtful and hopefully persuasive argument why, if you are a mail administrator, you may want to make a really important change to the way your mail server is configured. Or, if you are not a mail administrator, why you may want to forward a link to this blog entry to your mail administrator.
The main argument is simple. Mail administrators should:

delete all mail that you think contains a virus (virus false positives are practically unheard of);
bounce (actually - refuse delivery - it is up to the sending mail server to generate a bounce or not) mail that does not conform to mail protocol standards, cannot be delivered (user does not exist), or violates one of your policies (e.g. user quotas);
deliver all other mail to the intended recipient. Including mail that you think you are totally certain is spam.

Given the current day and age, you really really really should make an effort to warn your users if a message they are receiving may be spam, either by adding “SPAM*” to the subject line or another appropriate header to the message so they can choose to filter them out. This is Good and makes users happy.
Fortunately, most mail administrators do all of this. For all of you: my hat is off. Thanks for making the Internet a nice place to be.
Unfortunately, there are a number of mail administrators who have their mail servers configured to bounce messages back or drop them completely without feedback if they think they contain spam. Or worse, block all mail from a server that they think is a spammer.
This is a really really really bad idea.
Consider: if the message is from a spammer – bouncing generates yet more useless traffic clogging the Internet. Beside, that bounce will almost certainly generate a bounce back to you since the from address is almost always forged.
If the message is legit - then it shouldn’t be bounced!!
Some might argue the collateral damage position, which is: We need to let you know if you are using an Internet Service Provider that supports spammers. Your message isn’t spam, but it was sent from an evil ISP that has other clients that are spammers. You should pressure your ISP to stop supporting spammers which is the only way to rid the Internet of spamming companies.
There are several problems with this argument:

Is this your job? The year isn’t 1982 when email was a fun expirement. It’s 2006. People rely on email in ways that we, as mail administrators, can only guess. Really rely on it. Yes, we should be educating our users against relying on a technology that, when you understand how it works, should scare the daylights out of you. But there are better ways of doing that. Our job is to make sure that as close to 100% of messages sent our way get delivered to the proper user. Not lecture our users or any one else for that matter about who they should use as an ISP. Bouncing or dropping a message to essentially make a point is juvenile given the stakes.
It won’t work. There are a gazillion companies setup to send spam or poised to take over when there’s an opening. Telling a user who just learned how to send their first email message that they need to get a new ISP is just not a good strategy.

Another common argument is the system resources argument. We can’t possible deliver all spam without overwhelming our servers! We’re doing this to save our infrastructure!
If you are quietly sending all mail that you think is spam to /dev/null, then this argument makes sense. Of course, if this is true, then we really need to talk.
On the other hand, if you are bouncing suspected spam to the original sender, then you are really substituting one system resource for another. You are generating a new email message, which most certainly will bounce right back to you. Even if this is less expensive for your server, consider the traffic you generate for the rest of us. Also consider that: hard drives are cheap and without a lot of creativity you can auto delete messages from people’s spam folders that are older than x days.
If you are convinced (and I hope you are), please consider these policies:

Use your MTA to ensure proper mail protocol compliance. Period. Many MTA’s have very easy to use methods for auto rejecting mail that matches one or several black hole lists or mail that matches certain keywords. Don’t use these tests. If you currently use them, disable them. Your MTA’s job is to delivery mail, not make judgement calls. Your MTA may have mechanisms for making sure the server sending mail follows proper protocols (e.g. properly identifies itself with a fully qualified domain name in the HELO hand shake, etc.). This is Good. Bounce mail that doesn’t follow proper protocol. No arguments here. Just don’t make your MTA bounce mail based on tests that can potentially produce false positives.
Use a spam program to rank messages based on spam liklihood. You can use Spamassassin or Dspam or whatever you like. These programs use very complex and tested methods for evaluating the liklihood that a message is spam and then give it a ranking that your end users can use to filter out likely spam (but all mail is delivered). This way, if your spam program gets it wrong, they have the opportunity to either teach your spam filter how to get it right, or simply filter messages from a regular user to their inbox before the spam filter takes it out. This approach gives your users the power to control the spam filtering.

If the technical arguments aren’t convincing (or even if they are) you may also be interested in some very important political arguments why configuring your server this way is vitally important for the future of the  Internet, particularly the future of the Internet as an open mass communications medium.


Broadband wireless
Sun, 08 Jan 2006 09:32:44 +0000
The New America Foundation came out with an interesting report on broadband wireless. It’s a vital contribution toward the goal figuring out how we want to use the Internet. It’s great to see some forward action on the topic.
See the report here.


Certain Days
Sun, 08 Jan 2006 09:32:44 +0000
Mark your calendar. Or get a calendar to mark. May First/People Link member Certain Days is hosting their Freedom for Political Prisoners 2007 calendar launch party on Saturday, November 11: more info here.


Check out Mark Libkuman at the Labor Tech Conference in San Francisco
Sun, 08 Jan 2006 09:32:44 +0000
Mark Libkuman from May First/People Link and MFPL Member Open Flows will be presenting at the Labor Tech Conference November 17 - 19. He’s on Saturday, November 18 at 11:00 am on the “Web Sites, and Using Technology to Build Organizing and Information Networks” panel. Please support the conference and say hi to Mark if you can make it!


Electronic Recycling or Third World Dumping?
Sun, 08 Jan 2006 09:32:44 +0000
Thanks Prince for pointing out this article. Careful where you recycle - you just might be dumping your crap somewhere else.
The New York Times
October 24, 2005
Poor Nations Are Littered With Old PC’s, Report Says
By LAURIE J. FLYNN
Much of the used computer equipment sent from the United States to
developing countries for use in homes, schools and businesses is often
neither usable nor repairable, creating enormous environmental problems in
some of the world’s poorest places, according to a report to be issued today
by an environmental organization.
The report, titled “The Digital Dump: Exporting Reuse and Abuse to Africa,”
says that the unusable equipment is being donated or sold to developing
nations by recycling businesses in the United States as a way to dodge the
expense of having to recycle it properly. While the report, written by the
Basel Action Network, based in Seattle, focuses on Nigeria, in western
Africa, it says the situation is similar throughout much of the developing
world.
“Too often, justifications of ‘building bridges over the digital divide’ are
used as excuses to obscure and ignore the fact that these bridges double as
toxic waste pipelines,” says the report. As a result, Nigeria and other
developing nations are carrying a disproportionate burden of the world’s
toxic waste from technology products, according to Jim Puckett, coordinator
of the group.
According to the National Safety Council, more than 63 million computers in
the United States will become obsolete in 2005. An average computer monitor
can contain as much as eight pounds of lead, along with plastics laden with
flame retardants and cadmium, all of which can be harmful to the environment
and to humans.
In 2002, the Basel Action Network was co-author of a report that said 50
percent to 80 percent of electronics waste collected for recycling in the
United States was being disassembled and recycled under largely unregulated,
unhealthy conditions in China, India, Pakistan and other developing
countries. The new report contends that Americans may be lulled into
thinking their old computers are being put to good use.
At the Nigerian port of Lagos, the new report says, an estimated 500
containers of used electronic equipment enter the country each month, each
one carrying about 800 computers, for a total of about 400,000 used
computers a month. The majority of the equipment arriving in Lagos, the
report says, is unusable and neither economically repairable or resalable.
“Nigerians are telling us they are getting as much as 75 percent junk that
is not repairable,” Mr. Puckett said. He said that Nigeria, like most
developing countries, could only accommodate functioning used equipment.
The environmental group visited Lagos, where it found that despite growing
technology industries, the country lacked an infrastructure for electronics
recycling. This means that the imported equipment often ends up in
landfills, where toxins in the equipment can pollute the groundwater and
create unhealthy conditions.
Mr. Puckett said the group had identified 30 recyclers in the United States
who had agreed not to export electronic waste to developing countries. “We
are trying to get it to be common practice that you have to test what you
send and label it,” he said.
Mr. Puckett also said his group was trying to enforce the Basel Convention,
a United Nations treaty intended to limit the trade of hazardous waste. The
United States is the only developed country that has not ratified the
treaty.
Much of the equipment being shipped to Africa and other developing areas is
from recyclers in the United States, who typically get the used equipment
free from businesses, government agencies and communities and ship it abroad
for repair, sale or to be dismantled using low-cost labor.
Scrap Computers, a recycler in Phoenix, has eight warehouses across the
United States to store collected electronics before they are shipped to
foreign destinations, and Graham Wollaston, the company’s president, says he
is opening new warehouses at the rate of one a month. Mr. Wollaston, who
describes his company as a “giant sorting operation,” said there was a reuse
for virtually every component of old electronic devices: old televisions are
turned into fish tanks for Malaysia, and a silicon glass shortage has
created huge demand for old monitors, which are turned into new ones.
“There’s no such thing as a third-world landfill,” Mr. Wollaston said. “If
you were to put an old computer on the street, it would be taken apart for
the parts.”
Mr. Wollaston said the system was largely working, though he conceded that
some recyclers dump useless equipment in various developing nations, most
notably China. “One of the problems the industry faces is a lack of
certification as to where it’s all going,” he said. He says his company
tests all equipment destined for developing nations.
The Environmental Protection Agency concedes that “inappropriate practices”
have occurred in the industry, but said it did not think the problem should
be addressed by stopping all exports.
“E.P.A. has been working with the Organization for Economic Cooperation and
Development countries for the last several years on development of a program
that would provide much greater assurance that exports of recyclable
materials will be environmentally sound,” Tom Dunne, of the agency’s Office
of Solid Waste and Emergency Response, wrote in an e-mail message.

Copyright 2005 The New York Times Company



End of the Internet?
Sun, 08 Jan 2006 09:32:44 +0000
I saw an interesting article reposted on Portside. The original article, by Jeff Chester, warns of the threat we are facing because Internet Service Providers like Verizon and the cable companies will start restricting content. For example:

Consider what would happen if an online advertisement promoting nuclear power prominently popped up on a cable broadband page, while a competing message from an environmental group was relegated to the margins. It is possible that all forms of civic and noncommercial online programming would be pushed to the end of a commercial digital queue.
My question: what do you mean consider what would happen?? This has been going on since the beginning of media. I’m not sure this is really the best way to understand the threat. Consider instead: I use optimum online at home (provided by Comcast Cable). They already decided, in the name of stopping spam, that I am not allowed to send email directly from my laptop unless I use their email server (in other words, send all email through their servers),
Similarly, in the name of spam, entire providers (like all of AOL) regularly block all email sent from an entire server. This blockage is done both by humans, but more often by dumb software that has to be fixed by humans when it blocks the wrong server (which happens regularly).
Also… Chester talks a lot about the threat of tiers of service. I will be curious to see how broad open wifi networks affect the whole broadband connection market, especially considering the new wifi standars being experimented with that extent wifi access from hundreds of feet to miles.


Energy Solutions Conference: April 27 - 29
Sun, 08 Jan 2006 09:32:44 +0000
The Peak Oil NYC folks are planning a conference in late April that looks like it’s off to a great start. Now that Chevron and folks are jumping on the Peak Oil bandwagon, one of the greatest challenges will be preventing the issue from getting co-opted. Check out the conference - or the next Peak Oil NYC Meeting.


Free Software and Activists
Sun, 08 Jan 2006 09:32:44 +0000
Thanks Katrin for pointing out Bruce Byfield’s article asking why the free software movement is not closer to the activist movement. Certainly a topic we should be discussing. However, I don’t think the article covers what is fundamental: why do we see ourselves so distinctly as two movements? 
Seems to me that the problem is not getting Free software advocates to learn how to talk to activists and vice versa as if we are two distinct camps looking for common ground. Instead, the problem is recognizing that many free software technologists are activists and should recognize themselves and be recognized as leaders on this topic within the activist community. As far as I can tell neither is happening. As free software technologists (and all technologists for that matter) we often marginalize ourselves and allow others to marginalize us as mechanics of the movement with no vision to contribute. When we can recognize the Internet as a movement to be organized rather than a tool to be used, we may have a chance at overcoming this problem.


Grassroots Use of Technology
Sun, 08 Jan 2006 09:32:44 +0000
The conference, held on Saturday, June 17th, 2005 is hosted by the Organizers’ Collaborative based on Boston, MA. Please join us! We’ll be presenting two workshops on Spam and Activism and Domain Registration 101.



Great articles on Drupal
Sun, 08 Jan 2006 09:32:44 +0000
Developers interested in getting the basics of Drupal may be interested in a new series of articles written by IBM about Drupal.
In addition to an excellent overview of Content Management Systems and an explanation of why they chose Drupal, there’s a particularly good article on getting started with Drupal that covers some core concepts that will make your life a lot easier when developing a Drupal web site.
Check out the articles here.
If you are not a developer and want to know what all the Drupal talk is about - please see our Drupal FAQ.


How hot am I?
Sun, 08 Jan 2006 09:32:44 +0000
Knowing the temperature of your CPU is not something you need every day, but when you do need, well, it’s good to have the tools in place.
There are at least two ways to find out the temperature:

acpi - Advanced Configuration and Power Interface is a standard that defines common interfaces for hardware recognition, motherboard and device configuration and power management (see http://en.wikipedia.org/wiki/Acpi).

On a debian system:
apt-get install acpi

Then, have fun cat’ing files in /proc/acpi.
In particular, look for the “temperature” file in your /proc/acpi/thermal_zone/*/ directories. On my laptop I have THRC and THRS directories in my thermal_zone directory. THRC is the zone for my chip and THRS is the zone for the “system.” Not sure where the sensor for the “system” is placed. The point is - there are two thermometers (and your system might have less or more).
For reference purposes, I get the following when I cat my chip’s temperature zone:
jamie@liberace:~$ cat /proc/acpi/thermal_zone/THRC/temperature
temperature:             51 C

I like ACPI because it’s simple and provides information via an easy to navigate file system hierarchy.  Unfortunately, ACPI doesn’t work for all systems.
Consider poor chavez:
mayfirst@chavez:~$ cat /proc/acpi/thermal_zone/THRM/temperature
temperature:             0 C

I’m not sure where the THRM zone is, perhaps the M stands for the Moon (although I think it’s colder on the moon).
In any event, for chavez, we had to use lm-sensors
lm-sensors -  Lm-sensors is a hardware health monitoring package for Linux. It allows you to access information from temperature, voltage, and fan speed sensors. It works with most newer systems. (From apt-cache show lm-sensors).
Setting up lm-sensors involves:
apt-get install lm-sensors

Then, run the terrifying sensors-detect script as root to determine which kernel modules you need to insert in order for lm-sensors to work. The script spends so much time telling you that the commands you are about to execute are (probably) safe and won’t cause your system to crash that it makes you wish you were doing something safer, like brain surgery.
After running sensors-detect (if your system hasn’t crashed), you will be asked if you want the script to modify your /etc/modules file to include the new kernel modules on reboot. This is generally a good idea.
Then, run the sensors command (you don’t need to be root).
If all goes well, it should say something like this:
mayfirst@chavez:~$ sensors
w83627hf-isa-0290
Adapter: ISA adapter
VCore 1:   +1.36 V  (min =  +1.10 V, max =  +1.86 V)
VCore 2:   +3.28 V  (min =  +2.96 V, max =  +3.62 V)
+3.3V:     +3.36 V  (min =  +2.96 V, max =  +3.62 V)
+5V:       +5.19 V  (min =  +4.49 V, max =  +5.48 V)
+12V:     +12.04 V  (min = +10.76 V, max = +13.19 V)
-12V:     -11.62 V  (min = -12.77 V, max = -10.47 V)
-5V:       -3.09 V  (min =  -3.49 V, max =  -2.54 V)
V5SB:      +5.64 V  (min =  +4.87 V, max =  +5.94 V)
VBat:      +3.15 V  (min =  +2.82 V, max =  +3.46 V)
fan1:     6750 RPM  (min =  712 RPM, div = 8)                     (beep)
fan2:        0 RPM  (min =  712 RPM, div = 8)              ALARM
fan3:        0 RPM  (min =  712 RPM, div = 8)              ALARM
temp1:       +30Â°C  (high =   +50Â°C, hyst =   +45Â°C)   sensor = thermistor
temp2:     -48.0Â°C  (high =   +75Â°C, hyst =   +70Â°C)   sensor = thermistor
temp3:     +42.0Â°C  (high =   +75Â°C, hyst =   +70Â°C)   sensor = diode           (beep)
vid:      +1.388 V  (VRM Version 10.0)
alarms:
beep_enable:
Sound alarm enabled

And presto, you are up and running.
All too often, however, you will get:
jamie@liberace:~$ sensors
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.

instead.
If this happens, you should try upgrading to the most recent kernel. That’s not particularly helpful advice, but it’s the best I can do. And it might not work. Sorry.


How to write secure php mail scripts
Sun, 08 Jan 2006 09:32:44 +0000
Allowing people to fill out forms on your web site that generate email messages is an important part of any organizing web site. This feature allows you to have interested people contact you for more information, join an email list, or tell a friend about your site. PHP makes programming these types of forms a breeze.
Unfortunately, this ease of use also makes it a breeze to program these forms in ways that malicious users can take advantage of to use your web form to send email to anyone they want. Taking advantage of security holes often allows people to send advertisement to large numbers of people using your form!
Following are some PHP programming tricks which will help ensure that you don’t get used.
Whenever possible, hard code the “mail to” email address.
That means create a variable in your php code like:
$mail_to = 'your@emailaddress.org';

This technique is in contrast to placing that variable as a hidden variable in your form. When it is a hidden variable in the form, you open yourself up to malicious users manipulating the form so it posts their own “mail to” variable.
Sometimes you can’t hard code this value (like when you are creating a “tell a friend” page in which the user needs to submit the address of the user the email is being sent to). But for the forms in which the message is always sent to the same address, be sure to hard code it!
Validate all user input, especially email addresses.
Don’t trust anything the user sends you, especially email addresses.
It’s common to write forms in which a user puts in their email address and a message to send and then that message is emailed to us in such a way that is come “from” the user’s email address. Consider this code snippet:
$mail_to = 'your@emailaddress.org';
$mail_from = $_POST['mail_from'];
$subject = 'A message from a web surfer on our site';
$message = $_POST['message'];

$headers = "From: $mail_from";
mail($mail_to,$subjet,$message,$headers);

With this code, you very nicely and cleanly send a message someone posted to your site and magically it appears in your inbox coming from the user that sent it. This makes it easy to hit reply and have the reply go right to that person.
The problem is that many malicious users won’t submit just an email address in the form. They will submit an address, followed by a line break, followed by “Subject: Buy Viagra” followed by another line break followed by “To: another@emailaddress.org”, etc.
With the code snippet above, you are placing the entire unchecked contents of the mail_from field as headers in your email message! Yipes.
Avoid this problem by verifying that the from address really is a valid email address. I use the following snippet to do that:
// Define a constant that can be re-used when checking email
// addresses. If this is your first regular expression don't
// spend too much time looking at it. It may make you nauseous.
// (thanks to lightningbug.sf.net)
define(
	'EMAIL_MATCHER',
	'/^[^@\s]+\@(\[?)([-\w]+\.)+([a-zA-Z]{2,6}|[0-9]{1,3})(\]?)$/'
);
if(!preg_match(EMAIL_MATCHER,$mail_from))
{
	echo "Your email address does not appear valid.";
	exit;
}

With tell a friend emails - don’t allow users to write the body of the message.
Consider this code snippet:
$mail_to = $_POST['mail_to'];
$mail_from = $_POST['mail_from'];
$subject = "$mail_from thought you would like this site";
$message = $_POST['message'];
$headers = "From: $mail_from";
mail($mail_to,$subjet,$message,$headers);

Very nice way to allow a web surfer to send an email to their friends.
It’s also a very useful way to allow anyone in the world to send anyone else in
the world a completely arbitrary message. It’s trivial to write a program that
hits your web form repeatedly with a commercial message, each time with a new
email address.
There are many ways around this. The method I prefer is to allow users to type
their own personal message but limit it to 200 characters. Then, use a static
message that they can’t change which is automatically appended to the message.
This would allow a spammer to send a 200 character message to arbitrary
recipients, but the character limit is so low and the fact that your message is
appended to the bottom makes it much less attractive.
Here’s an example that uses this method:
$unchangeable_message = "Change this text to the message you want ".
"to be sent with all emails sent.";
$organization = "Your organization's name";
$copy_email = "your@emailaddress.org";
$errors = array();
$email_from = $_POST['email_from'];
$name = $_POST['name'];
$email_to = $_POST['email_to'];
$name = $_POST['name'];
$brief_note = $_POST['brief_note'];

define(
	'EMAIL_MATCHER',
	'/^[^@\s]+\@(\[?)([-\w]+\.)+([a-zA-Z]{2,6}|[0-9]{1,3})(\]?)$/'
);
if(!preg_match(EMAIL_MATCHER,$email_to))
{
	$message = "The address you entered for your friend does not ".
	"appear  to be valid. You entered $email_to.";
	$errors[] = $message;
}
if(!preg_match(EMAIL_MATCHER,$email_from))
{
	$message = "The address you entered for yourself does not ".
	"appear to be valid. You entered $email_from.";
	$errors[] = $message;
}
if($name == '')
{
	$errors[] = "Please enter your name.";
}
if(!preg_match("/^[a-zA-Z ']$/",$name))
{
	$message = "Your name can only contain letters, spaces and ".
	" apostrophes.";
	$errors[] = $message;
}
if(strlen($name) > 25)
{
	$errors[] = "Your name must be less than 25 characters.";
}
if(strlen($brief_note) > 200)
{
	$message = "You can only enter 200 characters for your note. " .
	"Sorry - this restriction is to prevent spammers from ".
	"abusing this form!";
	$errors[] = $message;
}
if(count($errors) > 0)
{
	echo "Thanks for your submission, however, we encountered the ";
	echo "following errors! Please hit back on your browser and try ";
	echo "again.";
	echo implode('&lt;br&gt;',$errors);
}
else
{
	$subject = "$name thought you'd be interested in $organization";
	$body = wordwrap($brief_note) . "\r\n\r\n--------\r\n\r\n" .
	wordwrap($unchangeable_message);
	$headers = "From: $email_from\r\nCc: $copy_email";
	if(mail($email_to,$subject,$body,$headers))
	{
		echo "Thanks for passing on the word about $organization";
	}
	else
	{
		echo "Woops. We hit a bug sending to $email_to. Please contact";
		echo "support.";
	}
}

Monitor
Be sure to watch your log files to make sure nobody is abusing your forms. If
you get strange emails sent by your form, forward them to support to make sure
it’s not a sign of abuse.


May First/People Blog blogged published in Social Policy Magazine
Sun, 08 Jan 2006 09:32:44 +0000
Alfredo Lopez’s blog on Spam and Censorship was published in the current issue of Social Policy Magazine. We’re excited to see the debate about the serious political implications of controlling spam reach an activist audience, rather than remaining soley an issue for techies.
To read the original blog and see more information about the May First/People Link Spam campaign, please see our Spam Campaign Page. Please check it out and then post a comment to let us know what you think.


PHP Form Mailer
Sun, 08 Jan 2006 09:32:44 +0000
Several weeks ago I submitted a blog for PHP developers about how to write
secure PHP email scripts. Recently, after some back and forth with the folks
working on the Sustainable South Bronx’s website, we decided
to package up that idea into a PHP mailer May First/People Link members can
simply download and use. Well, here it is! Please give it a shot and add
comments/make changes.
You can find it on the software page.


Proposal Review Software
Sun, 08 Jan 2006 09:32:44 +0000
During a meeting with a small NYC group that funds social justice organizations, we hit upon a need for software that can help manage the proposal review process. While there are gazillions of programs to help nonprofits manage their proposals and grants, there are much fewer that can help a foundation manage the proposal acceptance process.
Well, I take that back. There are probably thousands of programs that can do this, just precious few that are publicly available (proprietary or open source).
I started with google. What a mess. What search terms to use? Mostly garbage, but I did find the Grant Proposal Management System. Wow! That’s perfect. It meets all the needs we have and it’s fully open source. What a find. Except… there’s nothing to download and the SourceForge project page is empty. I’m still waiting to hear back from the email I sent two days ago.
Moving on, I did the obvious: ask people smarter than me. I posted a query to the Nonprofit Open Source Innitiative email list. This is a great list and I would encourage everyone interested in the topic to subscribe.
The responses were very interesting - including ideas that never occurred to me. Here’s a rundown with the current status of every idea:


Foundations that are doing something similar


Kurt remembered hearing about the Greater Kansas City Community Fund doing something similar. I little research turned up DonorEdge which is a great idea and has some overlap with what we’re looking for, but is not quite the same thing.


Katrin mentioned that the Meyer Memorial Trust in Portland was investigating this topic. Sure enough Marie got back to me saying that they are in the process of developing a full on open source application “that includes online proposal application all the way through reporting on outcomes, internal processing and reports, etc.  It also includes all the back end for managing the database.” She also pointed out that it is not yet ready (hence no link, etc.). Once it is ready, they will announce it far and wide.


Rich suggested I contact Philantech, saying he learned at the last NTEN conference that they were developing something similar. I have heard back from Dahna confirming this software project. According to Dahna “PhilanTrack is built with open source tools, though we are not currently open sourcing it, but rather offering it as a customizable hosted solution.” Check their web site or email info [at] philantech [dot] org for more information.


Something totally different: a lot of people suggested really good ideas that never occurred to me, including:

Request Tracker (RT) - RT is a piece of software used for tracking user requests, such as help tickets. As Mark pointed out, the work flow logic is very similar to reviewing proposals and RT is a stable program that could probably be configured to present different text.
In that vein Jon suggested salesforce.com - given all the contact management abilities of the program.
Jon also suggested Plone because it has a lot of built-in work flow features
Michael suggest using Open Journal Software - which is used for peer reviewing journal submissions.
Rolf joined in with a particular solution that he is implementing using Mantis. Mantis is similar to RT. He is actively working with a foundation and the core Mantis developers to abstract it enough so that it can be used with only configuration and template changes. This is very exciting.



Spam is out of control!
Sun, 08 Jan 2006 09:32:44 +0000
Especially the last few weeks we’ve seen an signficant rise in spam on the May First/People Link servers. We’ve started taking more aggressive evasive action, so you should see more spam identified with the SPAM subject line or rejected all together over the next few weeks. Thanks for your patience!
Keep in mind tips you can use to reduce spam. A good article on the subject from Tech Soup is available here.


The beginning of a frightning trend
Sun, 08 Jan 2006 09:32:44 +0000
According to ClickZ, AOL will be charging people to send bulk email to it’s user base. This is truly a frightening trend.
As spam has been accelerating, Internet Service Providers (ISPs) have been taking more and more desperate measures to protect their customers. One popular technique has been to block an entire server because, based on volume of mail and other inelegant algorithms, software running on the ISP’s servers decided that the server is sending spam.
Fortunately, AOL and other ISP’s have recognized that an enormous amount of legitimate email is blocked using this mechanism. Unfortunately, AOL has decided that the solution to this problem is that people sending legitimate email should pay.
Here at May First we host over 200 email lists, some with thousands of subscribers. Therefore, we inevitably send anywhere from 1,000 to 10,000 messages to AOL per day. All messages sent are commercial free communications between nonprofit organizations and their constituents, often carrying messages that will never be available in the mainstream media. Even if AOL charged 1/10th of one cent per email, May First could be billed over $150,000 per year just to send email to AOL.
If other big players in the ISP world pick up on this idea, it could effectively shut out huge parts of the Internet to non-commercial email content.


The Future of Political Email
Sun, 08 Jan 2006 09:32:44 +0000
I just did some quick and dirty tests on the 150,000 or so email addresses that are subscribed to the hundreds of email lists that we host. Somewhere between 30 and 40% of these email addresses end in either: hotmail.com, yahoo.com, aol.com, or (increasingly) gmail.com. Four providers - which happens to be consistent with the presentation that Chris at Democracy In Action gave at the AdvocacyDev conference.
If I were to place bets, I’d bet that the percentage will raise to about 50% in the next year as gmail really starts getting on a roll.
For many of us, this development is frightening in a rather abstract anti-corporation type of way.
However, considering the way spam is being treated, this development also happens to be frightening in a very specific and concrete way as well.
Many email administrators (including May First/People Link) rely on a tests to determine if a message may not be wanted by the recipient. We use a variety of tests against the body of the message and we check in with a couple databases on the network. If we think the recipient might not want it, we change the subject line to say SPAM so they can filter it out (but they always receive it).
AOL and Hotmail, however, are promoting a reverse approach. They are setting up preferred sender systems. In other words, if you pay money to a third party “certified email” company, then your email will be untagged and delivered to the user’s inbox regardless of any other checks.
AOL now has four ways  to get your email delivered to their users. Their whitelist and enhanced white list are part of their old strategies and are (at least for now free). Goodmail (which many people may remember from the Dear AOL campaign) is still there. Good Mail is one of several Certified Email vendors. You pay them to certify to the world that you are not sending email. Anyone who pays GoodMail (and passes whatever checks they do to ensure you are not a spammer - whatever that means to them) will get their email put straight and center into every AOL inbox they send to.
In a bow to nonprofits, AOL also offers a similar deal via Habeas and ISIPP to do virtually the same thing, but for free if you are a 501(c)(3).
Meanwhile, Hotmail has its own methods which include Sender Score Certified, yet another Certified Mail provider.
Granted, at this time, we activists without the budget to pay for these services, are, for the most part, still able to get our messages through. However, consider the Internet down the road, as spam proliferates and the big four continue dominating the percentage of email addresses in use by the people we are organizing. If just one or two of the big providers flipped a switch preventing mass email being delivered without certification it could effectively cripple political organizing on the Internet.
Sadly, this change is not happening due to big bad government stepping in a regulating an industry to benefit corporate America or even some kind of power move by a dominate company. This change is happening for two main reasons:

We are being seduced by “free” services
We are not applying any kind of critical thinking when it comes to the Internet in general and spam in particular - they way we do with other large scale social phenomenon.

If there is ever a time to start thinking politically about how we use the Internet - the time is now! More food for thought is available at our Spam Campaign page.


The only clients we talk about
Sun, 08 Jan 2006 09:32:44 +0000
Some of you have heard us correct you when you refer to yourself as our client (not client, member!).
Well, this blog is about a different kind of client: an email client (for abh’s definition of client - and there is no better, see below).
I got a support call from a member (and a suggestion from another member) - both of which have inspired this blog.
The member support call was about the particulars of our webmail program. The request went something along the lines of “In Yahoo, it does this - why can’t yours do that?” Well, the answer to that particular question was - “it can do that and let’s do it together so you know how.” However, after fielding these types of calls for years, I often get the sense of when a member really really really wants it her way. And this was one of those examples.
What I didn’t explain on the phone, is that if you want it your way (resisting the obvious joke), you really need a different email client then web mail. While we offer both SquirrelMail and IMP/Horde - two of the best webmail clients around, if you want to really have control over your email and fully integrate it into your computer, you may want an email client like <a href=-“http://www.mozilla.com/en-US/thunderbird/">Thunderbird .
Thunderbird installs on your computer. Depending on how you configure it, you can access all your email, even when you are not online. There are more powerful and faster ways to search and sort your email. And - you never have to worry about taking up too much space on our server.
Furthermore, May First/People Link fully support all the major email clients.
For more information on setting up an email client, please see our support pages.

So what’s a client? In the words of abh:
When you’re talking about computers and the internet, you’ll hear the word “server” all the time. You don’t hear “client” quite as often, but it is a good word to know. Internet services like web pages and email operate on a client-server system. Somewhere, out there, there is a server, a computer running software that allows it to take requests for information and pass out bits and bytes. To interact with a server, you have to have a “client” – software on your computer that knows how to read the information the server is passing out, and how to send information back to the server in a way that the server can handle. You use an email “client” to send and receive email, and to read and store it. Webmail programs like Horde and Squirrelmail are examples of clients, as are programs like Outlook, Eudora, Thunderbird and Kmail.


The pursuit of manliness harms men - and women as well
Sun, 08 Jan 2006 09:32:44 +0000
Thanks Rafa for this one.
Harvey Mansfield’s book Manliness hardly deserves mention except as a foil for talking about something much more interesting. In this case why men should support feminism out of self-interest (for those that can’t think of any other reasons).

…the dominant construction of masculinity in U.S. culture (rooted in aggression and a quest for power over others) leads not only to predictable injuries to women and girls (through the discriminatory practices and violence that stems from those values) but is also toxic to men. What creates risks for women also constrains the lives of men.
Check out the entire article here.


Trans Day of Justice
Sun, 08 Jan 2006 09:32:44 +0000
Join May First members Audre Lorde Project, Sylvia Rivera Law Project and others at the Trans Day of Justice Friday June 23 in New York City. For more information: www.fiercenyc.org.


Trend already over (for now)
Sun, 08 Jan 2006 09:32:44 +0000
AOL seems to be back tracking a bit on their original proposal. ClickZ reports that the unpaid White List will be kept after all. ClickZ attributes this change to the uproar not from independent Internet folks but instead from a competitor to the company that was going to provide the technology for the paid bulk sender program.
Although the biggest danger has passed for now, the fact that huge corporations like AOL, Yahoo and Gmail control such a huge number of email users combined with the way we talk about spam raises serious questions about how we use the Internet for political work.
Spam has really made a major impact in how we think about corporate intervention in our communications. If spam didn’t exist - we’d be up in arms over any attempt by a corporation to filter our mail. Thanks to spam, though, people actually want and expect their email to be filtered and are actually up in arms about getting too much unwanted email, expecting their corporate ISP to take care of it.
Granted spam is a problem - but it’s in our interests to keep the bigger picture in mind when we consider solutions to the problem.
Consider: There is no objective definition of spam. The technical word is “Unsolicited Bulk Email.” What does unsolicited mean?
If you go to a community meeting and sign in with your email address, are you soliciting email from the group collecting the sign up sheets? What if you signed up at the meeting and you don’t start receiving email for another 6 months at which point you’ve forgotten that you signed up? Many corporate ISP’s have a “Report message as SPAM” button - how many of their users really understand what that means? Is a message spam simply because a user identifies it as spam? How many users consider clicking the “Spam” button an easy way to make the message go away, even when they know they signed up for the list but don’t know how to unsubscribe or can’t be bothered to unsubscribe from it?
Or from an organizers perspective: What if you send a one time mass email to everyone who attended a conference you went to, using the email addresses published by the conference organizers? What if you collect email addresses in your database as part of your normal organizing efforts and periodically want to send mass email to these people about your activities?
According to most corporate approaches to spam - all the examples I’ve given are considered spam and would be blocked. According to definitions of spam in wide use - political organizing is spamming!


US Social Forum: Next regional meeting in December 9
Sun, 08 Jan 2006 09:32:44 +0000
The first United States Social Forum will happen in less than a year and the northeast is getting together to make concrete plans for organizing toward the USSF! The Next meeting will be on Saturday, December 9 at the Brecht Forum in New York City.
May First/People Link is heavily involved in making this forum a success - and we need your help. One way to learn more and get involved is to come to the next Northeast regional meeting, which is on Saturday, December 9, 2006 at the Brecht Forum. For more information - please click here. If you’d like to get involved, but you’re not sure how, feel free to contact support and we’d be happy to help.


What's a political techie?
Sun, 08 Jan 2006 09:32:44 +0000
I’ve been thinking recently about what it means to be a political techie, particularly around the conflicts between having a consulting relationship and a political relationship with the same “client” or “organization.”
For grounding purposes, let’s first consider what it means to be a political organizer using a classic example of a union organizer. For the sake of argument, let’s ignore the state of unions in the country today and instead focus on the methodology.
A union organizer walks into a non-union shop, where people are making very little money, have poor to zero job prospects outside of the particular company they are working for, and in many cases are the sole earner for a family. This union organizer’s job is to persuade them to organize, something that runs a very high risk of getting them fired. The organizer has to convince them to do something against their immediate self-interest, to take a major risk personally in order to get something much better in the long run not just for them, but for the entire group of people in the shop and for the global labor movement.
The union organizer will always be criticized for not caring about the individuals in the campaign (and many bad organizers really don’t care much about them). However, a good organizer, while caring about the individuals in the campaign, understands that the first priority is the bigger group, and by taking care of the bigger group (i.e. unionizing the shop), the individuals in the group will be better cared for in the long run.
What does this have to do with tech? Well, if we were to use this analogy to examine the role most tech consultants provide to their clients, it would be closer to the union organizer picking off the wealthiest of the workers and individually counseling them on how to get a raise or a promotion. And, the consultant would be paid by that individual worker.
Now, don’t go through the roof on this one. I’m not making this analogy to say there is no place for tech consultants, particularly in the social justice world. Given the reality of technology and the social justice movement, we’d be lost without all the enormous work tech consultants provide to individual organizations. This work is vital.
The point of the analogy is to demonstrate the conflict between providing individual support to organizations and doing political organizing for the entire sector.
So how does this play out?
It often plays out when we are asked for a software recommendation. Let’s take the office suite as an example.
As technologists, I think it’s hard to argue with the idea that we would all be best served by having an open document standard. In other words, a standard way for saving word processor files, spread sheet files, etc that is controlled by a body that is (at the least) semi-independent of any single corporation and (at the least) semi-democratic. With an open document standard, all the competing and various software programs that read and write office files could simply adopt one standard and voila: we have interoperability on a level we’ve never had in 25 years of office suites. Google, MS Office, OpenOffice, KWriter, the gazillions of groupware programs, and future programs we’ve never even dreamed of could all read and write the same type of file with a reasonable expectation of having the file look and behave the same way.
So, how do we organize our people to make that happen? Again, if politics is our guiding princicple, we organize a campaign with all the groups we work with to convince them to remove Microsoft Office from every computer within arms reach and install OpenOffice, which currently supports the best candidate for an open document standard.
Whoa! But that’s going to cause our fee-paying clients a lot of pain - they’re going to have to learn a new program, they’re going to have to deal with poor translation of files between their partners who are still using Microsoft Office, and they’re going to complain to me big time. Meanwhile, I’m going to have to work harder, explaining why this is a good idea. And, OpenOffice might not succeed, which will mean I’ve led them down the wrong path!! And all this means I might get fired!!
Yes. That’s true and that’s the conflict.
I’ve been in this game for a while, so I know that a lot of seasoned consultants will respond with: of course there is a conflict. That’s why the role of the consultant is not to make the decision for the client, but instead to layout the options and have the client make the decision. That’s what client empowerment is all about. Too often consultants tell the client what to do and that’s bad.
Agreed - telling a group what to do is bad. Forcing a group to remove MS Office and install OpenOffice is not a political or organizing move.
However, a union organizer does not walk into a non-union shop and say here are the pro’s and con’s of unionizing and here are the pro’s and con’s of working the way you’re working now. I’m going back to my hotel. Give me a call when you’ve made a decision.
While a good organizer makes clear what the dangers and pitfalls are of building a union, the organizer is there with a mission and a goal and, most importantly, a bigger vision of a better world. A traditional tech consultant is there (by virtue of paycheck) only to help with that single organization’s tech needs.
A big part of this conflict is funding based. While social justice movements need all the individual tech attention they can get, there are a lot of consultants out there. It’s an entire industry drawing thousands upon thousands of individuals, nonprofits and for profits all with the goal of meeting the individual needs of each nonprofit and getting re-imbursed by each individual nonprofit for the work accomplished.
In the late nineties, when the need for tech support to social justice organizations first surfaced, a different model was proposed, and in many places implemented. It was often referred to as the “Circuit Rider” model - with one techie managing a circuit of different nonprofits. The circuit rider was either funded by a foundation or the contributions of all the nonprofits receiving service. While this was the hot topic of the nineties, the concept has largely disappeared, mainly, I think people would argue, because it didn’t work.
I think that model will always fail if the premise is to provide individual consulting for each individual nonprofit. That relationship is better served by a consultant working just for you. However, it may be time to revive the circuit rider model for doing the political work. The social justice movement needs individual tech attention, but it also needs a broader political vision and the political leadership in the tech world to support it and move it forward.


Yahoo to Internet: Drop Dead
Sun, 08 Jan 2006 09:32:44 +0000
When you’re a giant on the Internet, like Yahoo, you really can make a difference. You can afford to show leadership, develop new ideas, and build and instracture that improves the Internet for everyone.
Or, you tell the world to drop dead.
That’s effectively what Yahoo starting doing some time last October. The web-o-sphere is full of complaints from people trying to send email to Yahoo that Yahoo refuses to accept, saying that it is being deferred (for the geeks: 421 Message from (209.51.172.11) temporarily deferred - 4.16.50). It continues to defer these messages, randomly letting them through according to the phases of the moon, the velocity of butterflies flapping their wings over the Sea of Japan or God knows what else.
In any event, we’re devoting yet more scarce resources ensuring that mail sent via our mail servers get delivered to Yahoo. The best we can hope for is that Yahoo’s backward ideas on how to ensure all mail gets delivered to their users will generate interesting conversations about how to choose an organization to entrust with your email.
For an interesting thread on the topic, check out AHFX’s blog.


A Real Racial Democracy?
Sat, 08 Jan 2005 09:32:44 +0000
NIKOLAS KOZLOFF writes:

As the war of words heats up between the Bush White House and Venezuelan president Hugo Chavez, the firebrand South American leader has boldly sought to forge ties with poor communities of color in the United States.
Chavez is extending the idea of a government for the poor across the border making it clear what Bush is not doing.
Read more.


Collectives
Sat, 08 Jan 2005 09:32:44 +0000
John Emerson writes an interesting overview of collectives with loads of good examples:
“Though we are supposedly living in a democracy, most of us spend our days working for private tyrannies. Living and participating in a democracy should consist of more than just voting once a year. We should be able to participate in the decisions that affect our lives.”
Read the article


Free Recycling this Saturday!
Sat, 08 Jan 2005 09:32:44 +0000
This saturday is the time to get rid of your junk, courtesy of the Lower East Side Ecology Center:
Dear Recyclers,
The Lower East Side Ecology Center is offering free electronic waste recycling events:
Brooklyn
Saturday, December 10th, 2005 at PS 321 in Park Slope at 180 7th Avenue look for a truck parked on 2nd Street just off 7th Avenue
Manhattan
Sunday, January 8th, 2006 at Union Square Park - North Plaza look for a truck parked in the northwest corner of the Plaza - 17th Street & Broadway
The events will be held rain or shine from 9 a.m. to 5 p.m
We will accept all working & non-working:
Computers/laptops, servers, mainframes
Monitors
Printers, scanners, fax-machines, cocpiers
Network devices (routers,hubs,modems, etc.)
Peripherals (keyboards,mice, cables, etc.)
TV’s, VCR’s & DVD Players
Audio visual devices
Radio/Stereo Equipment
Telecommunication (phones, answering machines, etc.)
Cell Phones, pagers, PDAs
Batteries
We would like to thank the New York Community Trust for supporting our electronic waste recycling program and PS 321, the PS 321PTA and the Department of Parks & Recreation for hosting these events.
Please help us spread the word, and let your friends & family know!!
If you like additional information please contact us at
www.lesecologycenter.org


From Bad to Worse
Sat, 08 Jan 2005 09:32:44 +0000
It was bad enough in 1994 when Congress passed the Wiretap law (Communications Assistance for Law Enforcement Act) which forces phone companies to install equipment in their premises that allow the federal government to tap the lines. Now, the law has been extended to universities, libraries and Internet Service Providers (like May First).
According to the NY Times, the universities are not concerned about the threat to our privacy since “the government would have to win court orders before undertaking surveillance.” Huh??
Two points of interest for May First Members:

May First will never comply with such a law.

It's irrelevant whether we do or not. We get our own service from an "upstream" provider who may already be complying with this order and spying on all traffic we send.
The conclusion: assume all Internet traffic is being monitored. If you want privacy use encryption! For more information about encryption:
Instruction for setting up encryption using Windows and the Mozilla Thunderbird email client
If you are using May First webmail - you can use encryption with SquirrelMail (go to the main login page, scroll down to the SquirrelMail link, login, click Option, then click GPG Plugin Options).
More Info about the Wiretap law:
Inside Bay Area
NY Times
Slash Dot
Wired


May First Moves into New Cabinet
Sat, 08 Jan 2005 09:32:44 +0000
Last week we moved all of our servers from the clutches of the Internet Channel (recently aquired by thorn.net) into our very own cabinet located at 25 Broadway in downtown Manhattan.
This move happened after months of being saddled with astronomical and questionably calculated overage charges by the Internet Channel (not to mention ridiculous termination fees and, at the last minute, $300/hour charge - 3 hour minimm - to hire a technician to unplug and hand over our servers when we moved).
What does this mean? It means that May First now has direct control over all of our servers. It means we have a stable-priced cabinet that can hold about 20 more servers, enabling us to easily expand in the future. And it means we have loads more bandwidth for high capacity sites, streaming video and more.


May First/People Link Party
Sat, 08 Jan 2005 09:32:44 +0000
Hi all - Please come to our Party on Wednesday, December 7 from 6:00 to 9:00 pm! For more information, check out the party details page.


Open Office gaining ground
Sat, 08 Jan 2005 09:32:44 +0000
Open Office (a free, open source office software suite) has been challenging Micrsoft Office (a bloated, overpriced office software suite) for a few years now. It just got a boost from google.
From the International Herald Tribune:

Google, the most-used Internet search engine, and Sun Microsystems said Tuesday that they would jointly promote Sun’s word processing and spreadsheet software in a direct challenge to Microsoft’s Office products.
Read more…


Save Public Access
Sat, 08 Jan 2005 09:32:44 +0000
Public Access Television is once again under attack. Even if you don’t regularly watch public access, consider the concept: if the cable companies want to lay cable under public streets and sidewalks, they should pay rent, right? And, what better way to pay rent than to provide the funding and channels to allow people to create their own television? It’s very good idea.
You can support by going to:
http://mnn.org/saveaccess


System Advisory
Sat, 08 Jan 2005 09:32:44 +0000
We are making a major upgrade to our server Wiwa over the weekend December 2 - December 3rd! Please see our advisories below for more information.
If you are having trouble accessing your email, and you typically enter your full email address as your username (e.g. info@mayfirst.org), try logging in with just the username portion of your email address instead (e.g. info).
Service Advisory: Directory Path Change
Service Advisory: Email Login Change


What does current working directory mean?
Mon, 01 Jan 0001 00:00:00 +0000
In computer terms, the current working directory (or just working directory)
refers to the directory or folder from where the running program or process is
operating. In a revision control system it refers to a local, working copy of
data - as opposed to the central, shared, or published copy of the data.
In the case of this site - it referes to the place to find out what I’m
currently working on.