Login

Repoze Blog

2010-09-24

Simon Pamiés BFG Talk and Tutorial Notes

Simon Pamiés recently released notes from a talk and a tutorial he gave at the 2010 Zope Conference in Dresden. Both the talk and the tutorial are about repoze.bfg.

posted at: 17:10 | permalink

2010-09-10

BFG: Speed Thrills

Charlie Clark's EuroPython 2010 presentation, entitled "BFG: Speed Thrills" is now online at http://bfg.repoze.org/videos#speedthrills .

posted at: 12:12 | permalink

2010-08-20

Brainrepublic on BFG

From the new-deployments-department: Brainrepublic is a newly launched website based on repoze.bfg and MongoDB. Congratulations to Andreas Jung.

posted at: 00:00 | permalink

2010-08-18

BFG at Froscon

Andreas Jung will be giving a talk (in German) on a highly scalable web application using BFG, MongoDB and RabbitMQ at Froscon in Sankt Augustin near Bonn in Germany on Sunday 22nd August. For more information see the Froscon website.

posted at: 13:12 | permalink

BFG at German Python Conference

At the German Python and Zope conference in Dresden in September Simon Pamies will be giving a tutorial on BFG (in German) on Tuesday 14th. He will also be giving an introductory talk on BFG at the conference. For more information see the conference website.

posted at: 00:00 | permalink

2010-08-11

Groundhog (Using BFG to Build a Microframework) #6: Events and Context Locals

In the sixth and final screencast in the Groundhog series, we allow our microframework's users to subscribe to and receive events, which are objects broadcast by BFG at various well-known points during its processing cycle. We also allow our users to define and send their own custom event types. Once we've built up some event machinery, we use it to implement "importable locals" (aka "stacked object proxies" or "context locals").

posted at: 16:17 | permalink

2010-08-10

Groundhog (Using BFG to Build a Microframework) #5: Exception Handling

In the fifth screencast in the Groundhog series, we equip our microframework with a way to translate exceptions raised by developers into HTTP responses. We then allow our developers to use convenience APIs named abort and redirect to raise exceptions. Finally, we allow them to customize the appearance of exceptions they raise.

posted at: 19:41 | permalink

2010-08-06

Groundhog (Using BFG to Build a Microframework) #4: Sessions and Flash Messages

In our fourth Groundhog screencast, we give our microframework's users a way to store information in sessions by making use of Beaker middleware. We then implement flash messaging on top of this sessioning machinery.

posted at: 20:55 | permalink

2010-08-05

Groundhog (Building a Microframework with BFG) #3: HTML Templating

In the third screencast in this series, we allow Groundhog users to render HTML using Jinja2 and Chameleon-ZPT templates. We add a render_template method of the Groundhog application object and we provide custom renderer globals.

posted at: 22:26 | permalink

2010-08-04

Groundhog (Building a Microframework with BFG) #2: URL Generation

In the second screencast in this series, we give Groundhog's users the ability to generate URLs based on routing patterns associated with user-defined view functions. At the end of the second video, a Groundhog user can use the url_for method of the application to generate URLs which point into his application. We also teach Groundhog how to redirect to slash-appended routes and we show off the interactive exception debugger.

posted at: 21:35 | permalink

Groundhog (Building a Microframework with BFG) #1: Mapping URLs to Code

In the first screencast in a series, we use BFG to begin building (with apologies to Armin Ronacher) a clone of the open source Flask "microframework" that we name Groundhog. At the end of the first video, we have built enough of Groundhog to allow users to map URLs to code in a way that will be familiar to Flask developers.

posted at: 00:25 | permalink

2010-05-07

WSGI Training at Plone Symposime East, 25 May 2010

Agendaless will be doing a WSGI training at Penn State's Business College on the third tutorial day of the PSE conference

Moonshining: Making WSGI for Fun and Profit

Abstract

The Python web development world has adopted on WSGI as an interoperability standard for Python web servers, applications, and middleware components. This tutorial examines that specification in detail, and shows practical examples of using WSGI to improve development productivity, integrate applications, and solve various deployment problems.

Topics

  • PEP 333: the WSGI spec
  • Using "stock" WSGI tools
    • paster
    • mod_wsgi
  • Extending the WSGI application
    • middleware
    • paste templates
    • paster configuration
  • Practical applications
    • Debugging errors with WSGI
    • Profiling / instrumenting a WSGI application
    • Unified authentication via repoze.who
    • Unified skinning via Deliverance

Duration

One day.

Cost

$150 per student. Includes a copy of the 'repoze.bfg' book.

posted at: 00:00 | permalink

Nice review of repoze.bfg

Low Kian Seong gives BFG a nice review:

As someone said long time ago to me ... all frameworks suffer from the same disease ... it can really do the 90% fast and great but it's the last 10% that is going to blast the skin from your bones ... okay so I am paraphrasing, but you get my point. So, I am going to put this one through the paces hopefully initially with a smallish project that is not so heavy with the funky specs with just one or two bumps on the road to stretch the framework a bit.

Let us know how your trial app progresses!

posted at: 00:00 | permalink

New BFG App: Pilot's Flight Log

posted at: 00:00 | permalink

2010-03-13

Jazkarta Blog: Sprinting at Pycon

Aaron VanDerlip from Jazkarta has a nice blog entry on his PyCon experience, including sprinting at the repoze.bfg table on porting one of his TurboGears applications to BFG.

Thanks, Aaron, for the props on the documentation!

As Aaron noted, it was great to get to work with so many talented people at PyCon.

posted at: 16:41 | permalink

2010-03-06

Zope.de Reviews BFG Book

Jan Ulrich Hasecke reviews the repoze.bfg book via the Zope.de website (English translation).

posted at: 00:00 | permalink

2010-03-05

Darryl Cousins' Codes Live with BFG

Darryl has put a screencast online of the demo session he is doing this evening at the Christchurch branch meeting of the New Zealand Python Users' Group:

Nice demo, Darryl

posted at: 00:00 | permalink

2010-02-25

PyCon 2010: BFG talks online

The PyCon 2010 audio-video team has done a really bang-up job getting the conference talks recorded, edited, and online quickly, The two BFG-relate talks are now available:

Tres Seaver's "Evolving Your Framework Under Fire" talk:

Carlos de la Guardia's "Pay Only For What You Eat" talk:

Enjoy!

posted at: 00:00 | permalink

2010-02-24

PyCon 2010: repoze.bfg Sprinting

Folks have been sprinting here at PyCon this week on a number of repoze-related projects:

BFG Cookbook

Andrew Sawyers, Chris Shenton, Shane Hathaway, and Reed O'Brien worked on an example "BFG Cookbook" application, intended to run in the Google App Engine cloud. They succeeded in indirecting the storage of the recipe entries, with working backends based on the GAE Big Table API, ZODB, Mongo, and an in-memory (nonpersistent) storage.

Code for the project is available on bitbucket:

http://bitbucket.org/sawdog/bfgcookbook/

BFG without Z's

Reed also worked on re-doing the Mongo-based URL shortener he wrote at last year's sprint, this time using the Ming bindings. He is working to have an example which uses none of the zope.* packaages, in order to show how BFG can be used in a variety of ways.

Chris McDonough and Chris Rossi have been sprinting with the Pylons and TurboGears folks on a package code-named "Marco." Marco moves some of the dispatching logic of BFG down into a package intended to be useful as the basis for a version of Pylons: if it succeeds, Pylons and BFG would become "personality" layers on top of the shared Marco library.

CMS Architecture

Tres Seaver has been working on a CMS architecture which he has been mulling for a long while now, code-named "Bonzai". The architecture decouples "asset repositories" from the "site layout" applications which consume them, imposing a RESTy service layer between the two.

Sprint Humor

In reaction to the perhaps overly cute mascots adopted by a certain other nameless framework, the BFG sprint team has adopted the "flying" or "war" pig as a mascot (not the cute Porky type, but a real tusker!).

http://static.repoze.org/war_pigs_sprint.png

posted at: 00:00 | permalink

2010-02-22

Running BFG 1.2 on Webfaction

After hearing Brandon Rhodes' talk on deploying applications to WebFaction, and talking with one of the WebFaction staff here at the PyCon 2010 sprints, I sprinted today on creating an installer script for creating a BFG webapp inside a WebFaction account.

To use it in your WebFaction account, create an application in the control panel using the "Custom Installer Script" option, and paste the URL of the script into the form. Once the installer finishes, you will have a "starter" application running in the webapp directory, You can start hacking on the code there, or upload your own app to it and tweak it there.

posted at: 00:00 | permalink

2010-02-16

PyCon 2010 talk: Evolving Your Framework Under Fire

Tres will be giving a talk at PyCon 2010 in Atlanta this Saturday at 2:15 PM. The talk, entitled Evolving Your Framework Under Fire covers the evolution of BFG as it was updated and extended to support the needs of KARL, a mission-critical knowledge management system commissioned by the Open Society Institute <http://soros.org/>.

The previous version of KARL was built on top of Plone, with more than four hundred communities and 75,000+ content pages. The application had evolved over several years, and had become increasingly difficult to scale and extend. In addition, OSI was actively recruiting participation in KARL from other, simliar organizations, whose own unique requirements would have to be folded into the mix.

Over the course of six months, a small team built the new version of KARL on top of the BFG framework, which we were simultaneously extending and elaborating to support the needs of KARL. During that period, we made more than forty releases of BFG, as well as releasing new versions of many supporting packages. The 1.0 release of BFG followed shortly on the successful launch of KARL in May, with subsequent versions continuing to be informed by the needs of OSI and the other KARL users.

This talk discusses the process of evolving the BFG framework, with particular emphasis on learning from the changes to the assumptions made by the framework at its inception. We discuss also the tradeoffs in the choice to include features in the framework versus leaving outside, either in the application or in supporting libraries. Finally, the talk proposes some general "lessons learned" which should be useful to Python programmers building any sort of framework.

Talk Outline

  • BFG's original motivation and design.
  • Overview of the requirements for KARL
  • Key changes to the BFG framework during the project:
    • adoption of documentation-driven development
    • replacing a number of the components which BFG initially depended on, with smaller or lighter-weight substitutes;
    • making other "standard" features (persistence / database integration) optional or replaceable;
    • removing performance-heavy "do-what-I-mean" features;
    • forcible converstion of text to unicode at all application boundaries;
    • adoption of a standard of 100% unit test coverage;
    • adding "url routing" as an option to "stock" object traversal.
    • enabling isolation of "customer-specific" policies and features from the core application code.
  • Issues encountered during this process
  • State of BFG at OSI's KARL launch
  • Subsequent evolution of BFG
  • Lessons learned

posted at: 00:00 | permalink

2010-02-10

BFG 1.2 Final Released

We're proud to announce the latest major release of the repoze.bfg web application framework, version 1.2.

It may be installed via:

easy_install -i http://dist.repoze.org/bfg/1.2/simple repoze.bfg

The most important feature addition in 1.2 is an "imperative" configuration mode. This implies:

  • A repoze.bfg application can now be contained within a single Python file.

  • Developers are no longer required to use or understand ZCML to create a

    repoze.bfg application (ZCML is, however still supported).

See the "What's New in 1.2" document at http://docs.repoze.org/bfg/1.2/whatsnew-1.2.html for a detailed list of new features, changes, and backwards incompatibilities.

The docs at http://docs.repoze.org/bfg/1.2 have been updated. Additionally, a printed book which includes the documentation has been published (see http://bfg.repoze.org/book). The documentation has been extensively reworked and restructured for the purposes of printing the book.

The 1.2 version of BFG replaces 1.1 as the "current" version, which means that the index URL http://dist.repoze.org/bfg/current/simple and the documentation URL http://docs.repoze.org/bfg/current both now point at the 1.2 versions of things. The older 1.1 index is available via http://dist.repoze.org/bfg/1.1/simple and the older 1.1 docs are available via http://docs.repoze.org/bfg/1.1.

Enjoy!

posted at: 00:00 | permalink

2010-02-04

BFG book available for purchase

The repoze.bfg Web Application Framework, Version 1.2" paperback is now available for purchase:

https://www.createspace.com/3422488

This is a high-quality printing of the along with extras such as a foreword from Paul Everitt, an author introduction, and cool cover artwork from the guys at Electrosoup (http://www.electrosoup.co.uk/). It is slightly over 500 pages long.

If you like repoze and BFG, please show your support by buying a copy!

Book abstract:

repoze.bfg is a small, fast, down-to-earth, open source Python web development framework. It makes real-world web application development and deployment more fun, more predictable, and more productive.

This book will show you how to develop web applications using repoze.bfg and Python step-by-step, including:

  • Using SQLAlchemy and ZODB along with repoze.bfg.
  • Using built-in templating facilities to render HTML and XML.
  • Using repoze.bfg security functionality to protect your application.
  • Exending an existing repoze.bfg application without changing its source code.
  • Using sessions within repoze.bfg.
  • Running repoze.bfg under mod_wsgi.
  • Deploying your repoze.bfg application to Google's App Engine.

This book is written by the primary author of repoze.bfg. It has developed along with framework itself, and therefore contains the most accurate and up to date information.

posted at: 00:00 | permalink

2010-01-29

A Few Links...

Darryl Cousins gives us a blog post explaining BFG on Google App Engine including test coverage support.

Srikanth Suri gives us a blog post showing how he has integrated BFG with repoze.catalog.

posted at: 00:00 | permalink

2010-01-22

Jarn Builds TribaSpace on BFG

Jarn has built Tribaspace using BFG and a collection of other technologies. The site is still in beta, but Tribaspace is a promotion site for fashion-related events. Looks good. ;-)

posted at: 00:00 | permalink

2010-01-21

Seantis releases BFG-based invoicing application

Fabian Reinhard from Seantis gmbh wrote to let us know that his company has released a BFG-based invoicing application:

First, thanks so much for your hard work on repoze!!
We are coming the long way from Zope/Plone and it's just amazing to play
with this new toy!

We just released a small app for invoicing: http://invoice.seantis.ch

Might be useful for other people out there since invoicing seems to be a
quite common use case! :-)

Check it out! It looks like a really nice application!

posted at: 00:00 | permalink

2010-01-05

BFG 1.2 on Jython

Eduardo Diaz blogs about running repoze.bfg on Jython. Given Eduardo did a lot of work to get it mostly-running, we'll try to make it a supported platform.

posted at: 00:00 | permalink

2009-12-30

Repozecast 5

Repozecast #5 released

http://static.repoze.org/casts/repozecast-5-20091230.mp3

Repozecast is an every-so-often podcast. This is the latest one, wherein Tres and I talk about BFG, the ZTK, and repoze.who.

Happy holidays!

  • Chris

posted at: 00:00 | permalink

2009-12-20

repoze.bfg 1.2a7 Released

Install via:

$ easy_install -U http://dist.repoze.org/bfg/1.2/simple repoze.bfg

The docs at http://docs.repoze.org/bfg/1.2 have been updated.

The changelog follows:

1.2a7 (2009-12-20)

Features

  • Add four new testing-related APIs to the repoze.bfg.configuration.Configurator class: testing_securitypolicy, testing_models, testing_add_subscriber, and testing_add_template. These were added in order to provide more direct access to the functionality of the repoze.bfg.testing APIs named registerDummySecurityPolicy, registerModels, registerEventListener, and registerTemplateRenderer when a configurator is used. The testing APIs named are nominally deprecated (although they will likely remain around "forever", as they are in heavy use in the wild).
  • Add a new API to the repoze.bfg.configuration.Configurator class: add_settings. This API can be used to add "settings" (information returned within via the repoze.bfg.settings.get_settings API) after the configurator has been initially set up. This is most useful for testing purposes.
  • Add a custom_predicates argument to the Configurator add_view method, the bfg_view decorator and the attribute list of the ZCML view directive. If custom_predicates is specified, it must be a sequence of predicate callables (a predicate callable accepts two arguments: context and request and returns True or False). The associated view callable will only be invoked if all custom predicates return True. Use one or more custom predicates when no existing predefined predicate is useful. Predefined and custom predicates can be mixed freely.
  • Add a custom_predicates argument to the Configurator add_route and the attribute list of the ZCML route directive. If custom_predicates is specified, it must be a sequence of predicate callables (a predicate callable accepts two arguments: context and request and returns True or False). The associated route will match will only be invoked if all custom predicates return True, else route matching continues. Note that the value context will always be None when passed to a custom route predicate. Use one or more custom predicates when no existing predefined predicate is useful. Predefined and custom predicates can be mixed freely.

Internal

  • Remove the repoze.bfg.testing.registerTraverser function. This function was never an API.

Documenation

  • Doc-deprecated most helper functions in the repoze.bfg.testing module. These helper functions likely won't be removed any time soon, nor will they generate a warning any time soon, due to their heavy use in the wild, but equivalent behavior exists in methods of a Configurator.

posted at: 00:00 | permalink

2009-11-28

repoze.bfg 1.2a1 released

repoze.bfg 1.2a1 is now released. It can be installed via:

easy_install -i http://dist.repoze.org/bfg/1.2/simple repoze.bfg

Or via PyPI.

Documentation for this release exists at http://docs.repoze.org/bfg/1.2/.

This is a major feature release. The new features of the release are detailed in a What's New document.

In particular, this release has a new "imperative configuration" mode that makes it possible to create the simplest repoze.bfg application as a single Python file.

posted at: 00:00 | permalink

2009-11-26

repoze.bfg 1.1.1 Released

Happy Thanksgiving folks.

repoze.bfg 1.1.2 has been released. Install it via:

easy_install -i http://dist.repoze.org/bfg/1.1/simple repoze.bfg

Or via PyPI.

It is a pure bugfix release. The changelog follows:

1.1.2 (2009-11-26)

Bug Fixes

  • When two views were registered with the same accept argument,

    but were otherwise registered with the same arguments, if a request entered the application which had an Accept header that accepted either of the media types defined by the set of views registered with predicates that otherwise matched, a more or less "random" one view would "win". Now, we try harder to use the view callable associated with the view configuration that has the most specific accept argument. Thanks to Alberto Valverde. (backport from trunk).

  • The ACL authorization policy debugging output when

    debug_authorization console debugging output was turned on wasn't as clear as it could have been when a view execution was denied due to an authorization failure resulting from the set of principals passed never having matched any ACE in any ACL in the lineage. Now in this case, we report <default deny> as the ACE value and either the root ACL or <No ACL found on any object in model lineage> if no ACL was found. (backport from trunk).

posted at: 00:00 | permalink

2009-11-15

repoze.bfg 1.1 released

The second major release of the BFG web framework (aka "repoze.bfg"), version 1.1, is now available.

For a list of changes made since the 1.0 release, see http://docs.repoze.org/bfg/1.1/whatsnew-1.1.html .

See http://bfg.repoze.org/ for other general information about repoze.bfg.

BFG is a Python web framework based on WSGI. It is inspired by Zope, Pylons, and Django.

BFG contains moderate, incremental improvements to patterns found in earlier-generation web frameworks. It tries to make real-world web application development and deployment more fun, more predictable, and more productive. To this end, BFG has the the following features:

  • WSGI-based deployment: PasteDeploy and mod_wsgi compatible.
  • Runs under Python 2.4, 2.5, and 2.6.
  • Runs on UNIX, Windows, and Google App Engine.
  • Full documentation coverage: no feature or API is undocumented.
  • A comprehensive set of unit tests which performs 100% statement coverage of the BFG codebase.
  • Sparse resource utilization: BFG has a small memory footprint and doesn't waste any CPU cycles.
  • Doesn't have an unreasonable set of dependencies: "easy_install"-ing repoze.bfg over broadband takes less than a minute.
  • Quick startup: a typical BFG application starts up in about a second.
  • Offers extremely fast XML/HTML and text templating via Chameleon (http://chameleon.repoze.org/).
  • Persistence-agnostic: use SQLAlchemy, "raw" SQL, ZODB, CouchDB, filesystem files, LDAP, or anything else which suits a particular application's needs.
  • Provides a variety of starter project templates. Each template makes it possible to quickly start developing a BFG application using a particular application stack.
  • Offers URL-to-code mapping like Django or Pylons' URL routing or like Zope's graph traversal, or allows a combination of both routing and traversal. This helps make it feel familiar to both Zope and Pylons developers.
  • Offers debugging modes for common development error conditions (for example, when a view cannot be found, or when authorization is being inappropriately granted or denied).
  • Allows developers to organize their code however they see fit; the framework is not opinionated about code structure.
  • Allows developers to write code that is easily unit-testable. Avoids using extraneous thread local data structures which hamper testability. Provides helper APIs which make it easy to mock framework components such as templates and views.
  • Provides an optional declarative context-sensitive authorization system. This system prevents or allows the execution of code based on a comparison of credentials possessed by the requestor against ACL information stored by a BFG application.
  • Behavior of an an application built using BFG can be extended or overridden arbitrarily by a third-party developer without any modification to the original application's source code. This makes BFG a good choice for building frameworks and other "extensible applications".
  • Zope and Plone developers will be comfortable with the terminology and concepts used by BFG; they are almost all Zope-derived.

posted at: 00:00 | permalink

Announcing the Release of BFG 1.1

repoze.bfg 1.1 has been released. You can install it via:

easy_install -i http://dist.repoze.org/bfg/current/simple repoze.bfg

Or via PyPI.

The "What's New In repoze.bfg 1.1" document details the changes made since the 1.0 release: http://docs.repoze.org/bfg/1.1/whatsnew-1.1.html

The docs at http://docs.repoze.org/bfg/1.1 have been updated.

The changelog from the prior 1.1b4 release follows:

1.1 (2009-11-15)

Internals

  • Remove dead IRouteRequirement interface from repoze.bfg.zcml

    module.

Documentation

  • Improve the "Extending an Existing Application" narrative chapter.
  • Add more sections to the "Defending Design" chapter.

posted at: 00:00 | permalink

2009-11-12

BFG 1.1b4 Released

repoze.bfg 1.1b4 has been released. You can install it via:

$ easy_install -i http://dist.repoze.org/bfg/dev/simple repoze.bfg

Or via PyPI.

The docs at http://docs.repoze.org/bfg/1.1 have been updated.

The changelog follows:

1.1b4 (2009-11-12)

Bug Fixes

  • Use alsoProvides in the urldispatch module to attach an

    interface to the request rather than directlyProvides to avoid disturbing interfaces set in a NewRequest event handler.

Documentation

  • Move 1.0.1 and previous changelog to HISTORY.txt.
  • Add examples to repoze.bfg.url.model_url docstring.
  • Add "Defending BFG Design" chapter to frontpage docs.

Templates

  • Remove ez_setup.py and its import from all paster templates,

    samples, and tutorials for distribute compatibility. The documentation already explains how to install virtualenv (which will include some setuptools package), so these files, imports and usages were superfluous.

Deprecations

  • The options kw arg to the repoze.bfg.router.make_app

    function is deprecated. In its place is the keyword argument settings. The options keyword continues to work, and a deprecation warning is not emitted when it is detected. However, the paster templates, code samples, and documentation now make reference to settings rather than options. This change/deprecation was mainly made for purposes of clarity and symmetry with the get_settings() API and dicussions of "settings" in various places in the docs: we want to use the same name to refer to the same thing everywhere.

posted at: 00:00 | permalink

2009-11-02

Cobbler's Children Shod?

We have just rolled out a new version of the Repoze blog, based on BFG.

After a couple of years of using SVN to waldo in content to the third-party blogging code, we decided to eat some dogfood. Tres' son, Will, put together the blog software, including a script to migrate the existing content, preserving all existing URLs and dates.

Enjoy!

posted at: 00:00 | permalink

2009-09-09

New bfg.repoze.org Website

Thanks to ElectroSoup, the BFG website has a new design. The design is excellent; ElectroSoup does fantastic work should you need some design help.

posted at: 00:00 | permalink

2009-07-15

repoze.urispace, repoze.dvselect, and Plone, oh my!

After being the guinea pig to iron out bugs and documentation issues for the pairing of repoze.urispace and repoze.dvselect, Eric Brown went above and beyond the call of duty by writing up how to use the pair to theme multiple Plone sites using Deliverance.

posted at: 00:00 | permalink

2009-07-10

RepozeCast #4

RepozeCast #4 has hit the streets, wherein Tres and Chris discuss mostly pointless Repoze-related stuff.

posted at: 00:00 | permalink

2009-06-05

repoze.who / repoze.what Featured in Python Magazine

Gustavo Narea reports:

A few months ago I wrote an article on repoze.who and repoze.what, which has just been published on Python Magazine: http://pymag.phparch.com/c/issue/view/98.

I believe it's a good resource for those who are new to both frameworks (even if they aren't familiar with WSGI or auth in general yet), as well as for current users to better understand how repoze.who/what work and so make the most out of them.

posted at: 00:00 | permalink

2009-06-04

repoze.urchin 0.1 release

I just whipped up a tiny bit of middleware for OSI's version of the KARL project: they needed to put markup for Google Analytics into each page served. We wanted to avoid adding anything to the core KARL software which imposed such a policy on other users of KARL; we might have added it via the "customization package" for OSI, but it would have still required changing the core software in ways that seemed too invasive.

This seemed like a natural use for WSGI middleware, which could intercept the outbound respond and add the additional markup, without requiring any changes to the application. It turns out that the implementation is simple, cheap and generic: it would be useful for any WSGI-based deployment which wants to use Google Analytics.

So, I released it to the Cheeseshop, with the minimal docs also served from there.

Enjoy!

- Tres Seaver

posted at: 00:00 | permalink

2009-05-27

Register now for the BFG Tutorial at Plone Symposium East 2009

We will be presenting a half-day tutorial on repoze.bfg at the 2009 Plone Symposium East, hosted by the awesome WebLion group at Penn State. The tutorial is a hands-on (bring a laptop, and be ready to code) introduction to developing applications using BFG.

The tutorial runs on Thursday, 26 May 2009 (exact times TBD). The cost for the tutorial is $75.00. Register now!

Paul will also be talking about an ongoing BFG customer project, and how it relates to Plone. This talk starts at 11:00 AM on Friday, 28 May 2009.

Hope to see you there!

posted at: 00:00 | permalink

2009-04-30

Six Feet Up on Integrating Zine and Plone Using repoze.zope2 and Deliverance

Calvin Hendryx Parker of Six Feet Up writes a blog entry about integrating the Zine blog engine together with Plone and Deliverance, using repoze.zope2. The blog entry itself is hosted on that setup.

posted at: 00:00 | permalink

2009-03-23

Sprints at PyCon 2009

We are lookking forward to working with a number of folks at the sprints following Pycon this year.

There are seventeen folks signed up to date to work on Pylons, Repoze, TurboGears, and the other WSGI-centric Python web freamworks. Once exciting opportunity is to have a lot of cross-pollination between the frameworks, looking for ways to work together and share more code.

posted at: 00:00 | permalink

2009-03-13

Mikko Ohtamaa: Setting up Plone 3.2 and Repoze, hackyish

Mikko Ohtamaa provides a blog entry about how to set up Plone 3.2 under repoze.zope2.

posted at: 00:00 | permalink

2009-02-05

Carlos de la Guardia: Screencast, Plone + repoze.bfg

Once again, kudos to Carlos for his screencast on using Plone in conjunction with repoze.bfg.

Carlos writes that the BFG version, which publishes content mirrored to an RDBMS from the Plone CMS backend, renders pages at around 100x faster than Plone itself.

He is hoping to release the software, developed in conjunction with a big project for the Chilean Library of Congress, in time for PyCon, where he will be presenting about the topic. Way to go, Carlos!

posted at: 00:00 | permalink

2009-01-19

Repozecast 3: Wherein We Cut It Short

Repozecast 3 (mp3), a podcast about Repoze, is very short; less than 10 minutes. It talks about repoze.bfg. The reason it's so short is that we talked for another forty-five minutes after the Handy Zoom recorder had run out of space. We'll try to recover the ground we lost in subsequent epsisodes. Have fun!

- Chris McDonough

posted at: 00:00 | permalink

repoze.what 1.0 Final Released!

The repoze.what authorization framework has its first stable release!

repoze.what, which is the default authorization framework in TurboGears 2, was initially a TurboGears-specific repoze.who plugin (tg.ext.repoze.who) to support authorization based on the groups the authenticated user belongs to and the permissions granted to such groups, written by Chris McDonough, Florent Aide and Christopher Perkins.

The plugin evolved as an framework for arbitrary WSGI applications which allows developers to store the groups and permissions of the application in other source types (not only databases), just to name a few of the features implemented as a TurboGears-independent project.

The code sample below illustrates how this fully documented and tested framework (yes, its code coverage is at 100%) can be used:


# Sample use in TurboGears 2; pay attention to the line with the "@require"

class RootController(BaseController):

    # ...



    @expose('algo.templates.index')

    @require(predicates.has_permission('manage', msg=_('Only for managers')))

    def manage_permission_only(self):

        return dict(page='managers stuff')

In the example above, only people with the "manage" permission will be granted access to the "manage_permission_only" action. Also, if access is denied (i.e., user doesn't have the "manage" permission), she will be redirected to the login form and the message "Only for managers" will be displayed; a behavior that is fully customizable.

This groups/permissions-based authorization pattern is just the default pattern supported in repoze.what, and you can extend it to support your own pattern by creating so-called "predicates".

Planning for the upcoming major release of the package has already started, so please don't hesitate to report the features you want to see in this release!

Special thanks go to Chris McDonough for his support throughout the development of repoze.what.

-- Gustavo Narea.

posted at: 00:00 | permalink

Pycon US 2009: The Big F'n Tutorial

Pycon US 2009 is in Chicago this year again. I love going there. This year at the conference, I'll be presenting a tutorial on the repoze.bfg web framework along with Chris Perkins. The tutorial is on Thursday, March 26. If you're interested in repoze.bfg, this is a great way to learn how it works.

Chris Perkins will also be presenting a tutorial on ToscaWidgets: Test Driven Modular Ajax on the same day.

- Chris McDonough

posted at: 00:00 | permalink

2009-01-14

Carlos de la Guardia Takes up the "Songlist" Meme

Kudos to Carlos for his writeup on implementing the "songlist" app, now circulating in the Python blogosphere, using repoze.bfg.

Carlos writes that the BFG version, implemented in ~50 lines of Python and ZCML, benchmarks at 1060 requests per second, compared to the 1780 r/s of the "raw" WSGI version on the same machine.

posted at: 00:00 | permalink

2008-11-13

Using repoze in a Plone buildout

Props to Tom Gross, who has tackled the job of figuring out and documenting the minimal set of changes needed to make use of repoze and WSGI from within a stock Plone buildout.

Correction: Tom Gross sent me the link, but it was to a blog post by Tim Terlegård. Apologies for the mistake!

posted at: 00:00 | permalink

2008-10-20

Plone Conference Presentation on repoze.bfg

I've put the slides up of the presentation I gave at the 2008 Plone Conference about repoze.bfg (named "repoze.bfg: A Zope explosion). Enjoy!

-Chris

posted at: 00:00 | permalink

2008-10-04

repoze.squeeze and repoze.bitblt middleware

Malthe Borch and Stefan Eletzhofer have developed some nice middleware under the "repoze" flag. Malthe blogs about two of these creations: repoze.squeeze and repoze.bitblt.

posted at: 00:00 | permalink

A Tour of repoze.catalog

All versions of Zope depend heavily on the Zope Object Datbase (ZODB). Because ZODB is less a database and more a persistent object store (it doesn’t possess a query language; Python is its query language), its been necessary to create indexing and searching facilities for data stored in ZODB.

The first iteration of searching and indexing for ZODB-based applications (at least post-Principia, which had something named Tabula, which I never actually used) was the ZCatalog. The ZCatalog was entirely tied to Zope2, and still remains in heavy use today within Zope 2 applications such as Plone.

The second iteration was zope.app.catalog, which was a ZCatalog do-over for Zope 3 applications.

Neither of these searching and indexing packages are particularly easy to use outside of a Zope application. Each makes various assumptions about the content objects that need indexing or the environment that aren’t appropriate for arbitrary applications. For instance, ZCatalog wants objects you want to catalog to have a getPhysicalPath method which returns a “path”. An instance of zope.app.catalog makes the assumption that that it’s located within a Zope 3 “site” object within a ZODB, and assumes that you want query result sets to be sets of Python references to the original object you indexed. In other words, these packages assume too much to be maximally useful outside the context in which they were developed. Repoze <http://repoze.org> is a project which has as a stated goal making it easier for non-Zope Python developers to use Zope technologies outside Zope, so this seemed like a natural thing to do under the Repoze flag.

repoze.catalog borrows heavily from zope.app.catalog and depends wholly on the zope.index package for its index implementations, but assumes less about how you want your indexing and querying to behave. In this spirit, you can index any Python object; it needn’t implement any particular interface except perhaps one you define yourself conventionally. repoze.catalog does less than any of its predecessors, in order to make it more useful for arbitrary Python applications. Its implemented in terms of ZODB objects, and the ZODB will store the derived index data, but it assumes little else. You should be able to use it in any Python application. The fact that it uses ZODB is ancillary: it’s akin to Xapian using “flint” or “quartz” backends.

Indexing

To perform indexing of objects, you set up a catalog with some number of indexes, each of which is capable of calling a callback function to obtain data about an object being cataloged:

from repoze.catalog.indexes.field import CatalogFieldIndex

from repoze.catalog.indexes.text import CatalogTextIndex



from repoze.catalog.catalog import Catalog



def get_flavor(object, default):



    return getattr(object, 'flavor', default)



def get_text(object, default):



    return getattr(object, 'text', default)



catalog = Catalog()



catalog['flavors'] = CatalogFieldIndex(get_flavor)

catalog['text'] = CatalogTextIndex(get_text)



Note that get_flavor and get_text will be called for each object you attempt to index. Each of them attempts to grab an attribute from the object being indexed, and returns a default if no such attribute exists.

Once you’ve got a catalog set up, you can begin to index Python objects (aka “documents”):

class IceCream(object):



    def __init__(self, flavor, description):

        self.flavor = flavor



        self.description = description



peach = IceCream('peach', 'This ice cream has a peachy flavor')



catalog.index_doc(1, peach)



pistachio = IceCream('pistachio', 'This ice cream tastes like pistachio nuts')



catalog.index_doc(2, pistachio)

Note that when you call index_doc, you pass in a docid as the first argument, and the object you want to index as the second argument. When we index the peach object above we index it with the docid 1. Each docid must be unique within a catalog; when you query a repoze.catalog catalog, you’ll get back a sequence of document ids that match the query you supplied, which you’ll presumably need to map back to the content object in order to make sense of the response; you’re responsible for keeping track of which objects map to which document id yourself.

Querying

Once you’ve got some number of documents indexed, you can perform queries against an existing catalog. A query is performed by passing keyword arguments to the search method of the catalog object:

catalog.search(flavor=('peach', 'peach'))



Each keyword argument passed to search is either the name of an index contained within the catalog or a special value that specifies sort ordering and query limiting. In the above example, we specified no particular sort ordering or limit, and we’re essentially asking the catalog to return us all the documents that match the word ‘peach’ as a field within the field index named flavor. Each index specifies its own query argument style: field indexes specify that the value you must pass in be a “range” search, so we pass in the tuple ('peach', 'peach'), which can be read “from peach to peach”. Other types of indexes accept different query parameters. For example, text indexes accept only a term:

catalog.search(description='nuts')

The result of calling the search method is a two tuple. The first element of the tuple is the number of document ids in the catalog which match the query. The second element is an iterable: each iteration over this iterable returns a document id. The results of catalog.search(description='nuts') might return:

(1, [2])

The first element in the tuple indicates that there is one document in the catalog that matches the description ‘nuts’. The second element in the tuple (here represented as a list, although it’s more typically a generator) is a sequence of document ids that match the query.

You can combine search parameters to further limit a query:

catalog.search(flavor=('peach', 'peach'), description='nuts')



This would return a result representing all the documents indexed within the catalog with the flavor of peach and a description of nuts.

Index Types

Out of the box, repoze.catalog supports three index types: field indexes, keyword indexes and text indexes. Field indexes are meant to index single discrete values, and queries to a field index will only match if the value which was indexed matches the query exactly. Keyword indexes are essentially field indexes which index sequences of values, and which can be queried for any of the values in each sequence indexed. Text indexes index text using the zope.index.text index type, and can be queried with arbitrary textual terms. Text indexes can use various splitting and normalizing strategies to collapse indexed texts for better querying.

Helper Facilities

repoze.catalog provides some helper facilities which help you integrate a catalog into an arbitrary Python application. The most obvious is a FileStorageCatalogFactory, which makes it reasonably easy to create a Catalog object within an arbitrary Python application. Using this facility, you don’t have to know anything about ZODB to use repoze.catalog. If you have an existing ZOB application, however, you can ignore this facility entirely and use the Catalog implementation directly.

More Info

For more information, see the repoze.catalog documentation.

posted at: 00:00 | permalink

2008-09-26

repoze.bfg 0.3.8 Released with Chameleon and Genshi-style Support

Malthe Borch has been toiling away at renaming and refactoring the z3c.pt templating engine into something he's calling Chameleon. The first releases of Chameleon were made a couple of days ago, and I've updated repoze.bfg to a 0.3.8 version that uses Chameleon instead of z3c.pt. You can get this version by using easy_install -i http://dist.repoze.org/lemonade/dev/simple -U repoze.bfg.

repoze.bfg no longer supports z3c.pt itself, but all apps that use the repoze.bfg.template API functions will continue to function properly, albeit using Chameleon "under the hood" instead of z3c.pt. Importing API functions from repoze.bfg.template is now a deprecated spelling (although these APIs will remain around for a good long time, and won't be removed without a deprecation warning being issued). Newly generated projects will by default import template-related functions from repoze.bfg.chameleon_zpt instead of repoze.bfg.template.

One of the things that moving to Chameleon allows us to do easily is to support Genshi-style templates. To that end, there are new APIs in repoze.bfg which allow you to use Genshi-style templates out of the box. See the API documentation regarding the repoze.bfg.chameleon_genshi module for more info.

Thanks Malthe!

- Chris McDonough

posted at: 00:00 | permalink

2008-09-17

Christian Scholz: repoze.bfg and ToscaWidgets and storm and repoze.who.openid, oh my!

I enjoyed talking with Christian at the DZUG Tagung last week about how he was working with various repoze pacakges. It turns out that he has been blogging about them too:

Very nice work!

posted at: 00:00 | permalink

2008-09-12

DZUG Presentation Slides

Slides from Tres' presentation and lightning talk at the 2008 DZUG Tagung are now online:

posted at: 00:00 | permalink

2008-09-02

Repoze Presentation at ZPUG-DC Tuesday September 2, 2008

I (Chris McDonough) will be making a presentation about the new repoze.bfg Python web framework at a meeting of the Washington DC Zope and Python User's Group on Tuesday, September 2 2008 in the Radio Free Asia boardroom. Come on down if you're around!

For more information, see the ZPUG DC announcement.

posted at: 00:00 | permalink

2008-08-28

repoze.bfg incompatibility in 0.3.4

I've just released repoze.bfg 0.3.4, which contains a backwards incompatible change (the first since 0.1).

From the README.txt


  - Make ``repoze.bfg`` a namespace package so we can allow folks to

    create subpackages (e.g. ``repoze.bfg.otherthing``) within

    separate eggs.  This is a backwards incompatible change which

    makes it impossible to import "make_app" and "get_options" from

    the ``repoze.bfg`` module directly.  This change will break all

    existing apps generated by the paster code generator.  Instead,

    you need to import these functions as

    ``repoze.bfg.router:make_app`` and

    ``repoze.bfg.registry:get_options``, respectively.  Sorry folks,

    it has to be done now or never, and definitely better now.

I hope this doesn't cause anyone too much inconvenience.

posted at: 00:00 | permalink

2008-08-24

Carlos' repoze.shootout In Action

Carlos de la Guardia came up with an example "idea competition" application named repoze.shootout a week or two ago. You can now see it "in action" at http://shootout.repoze.org. It's purty these days.

repoze.shooout is a repoze.bfg sample application which currently demonstrates URL dispatch, integration with SQLAlchemy, integration with the Deliverance theming engine and integration with the repoze.who WSGI authentication system.

You can check this application out of Subversion via:


  svn co http://svn.repoze.org/repoze.shootout/trunk repoze.shootout

Or just browse it via http://svn.repoze.org/repoze.shootout/trunk

posted at: 00:00 | permalink

2008-08-11

Carlos de la Guardia: repoze.shootout

Carlos de la Guardia has checked in a cool demo application for repoze: repoze.shootout is an example "idea competition" application, demonstrating URL dispatch and integration with SQLAlchemy.

Check this application out of Subversion via:


  svn co http://svn.repoze.org/repoze.shootout/trunk repoze.shootout

It turns out the application also integrates repoze.who. Brilliant!

posted at: 00:00 | permalink

2008-07-31

Using repoze.tm2 with Pylons and Storm

Olaf Conradi just posted a good tutorial on using repoze.tm2 with Pylons and the Storm object relational mapper. Check it out!

posted at: 00:00 | permalink

2008-07-01

Repozecast Number 2

Repozecast #2, Plone Symposium Wrapup

The second episode (mp3) is now up. It contains ramblings about Repoze and how it relates to:

  • A wrapup of the Plone Symposium in New Orleans 2008
  • The current state of Zope-related Repoze software
  • Info about repoze.who
  • Recent Zope/Plone/Python user group talks
  • Thoughts about Deliverance

Enjoy!

-Chris

posted at: 00:00 | permalink

2008-06-10

repoze.who 1.0 Released

Version 1.0 of the repoze.who WSGI authentication framework has been released. You can get it via easy_install -i http://dist.repoze.org/who/latest/simple repoze.who.

Version 1.0 has optional support for middleware configuration via a config file (thanks to Tres). Being a framework, repoze.who is configuration-heavy, and it can provide a better separation of concerns and more convenience to wire it up via a configuration file than via Python code. So rather than configuring the middleware and attendant plugins via straight Python code, you can now wire who configuration up in an .ini file:


    # who.ini



    [plugin:form]

    # identification and challenge

    use = repoze.who.plugins.form:make_plugin

    login_form_qs = __do_login

    rememberer_name = cookie

    form = %(here)s/login_form.html



    [plugin:auth_tkt]

    # identification

    use = repoze.who.plugins.auth_tkt:make_plugin

    secret = s33kr1t

    cookie_name = oatmeal

    secure = False

    include_ip = False



    [plugin:basicauth]

    # identification and challenge

    use = repoze.who.plugins.basicauth:make_plugin

    realm = 'sample'



    [plugin:htpasswd]

    # authentication

    use = repoze.who.plugins.htpasswd:make_plugin

    filename = %(here)s/passwd

    check_fn = repoze.who.plugins.htpasswd:crypt_check



    [plugin:sqlusers]

    # authentication

    use = repoze.who.plugins.sql:make_authenticator_plugin

    query = "SELECT userid, password FROM users where login = %(login)s;"

    conn_factory = repoze.who.plugins.sql:make_psycopg_conn_factory

    compare_fn = repoze.who.plugins.sql:default_password_compare



    [plugin:sqlproperties]

    name = properties

    use = repoze.who.plugins.sql:make_metadata_plugin

    query = "SELECT firstname, lastname FROM users where userid = %(__userid)s;"

    filter = my.package:filter_propmd

    conn_factory = repoze.who.plugins.sql:make_psycopg_conn_factory



    [general]

    request_classifier = repoze.who.classifiers:default_request_classifier

    challenge_decider = repoze.who.classifiers:default_challenge_decider



    [identifiers]

    # plugin_name;classifier_name:.. or just plugin_name (good for any)

    plugins =

          form;browser

          auth_tkt

          basicauth



    [authenticators]

    # plugin_name;classifier_name.. or just plugin_name (good for any)

    plugins =

          htpasswd

          sqlusers



    [challengers]

    # plugin_name;classifier_name:.. or just plugin_name (good for any)

    plugins =

          form;browser

          basicauth



    [mdproviders]

    plugins =

          sqlproperties

Then you can use a constructor to create the configuration based on the .ini file, e.g.:


from repoze.who.config import WhoConfig



app = {next app in pipeline}

here = os.path.dirname(__file__)

config_file = os.path.join(here, 'who.ini')

parser = WhoConfig(here)

parser.parse(open(config_file))

middleware = PluggableAuthenticationMiddleware(app,

             parser.identifiers,

             parser.authenticators,

             parser.challengers,

             parser.mdproviders,

             parser.request_classifier,

             parser_challenge_decider,

             log_stream = sys.stdout,

             log_level = logging.DEBUG,

             )

There is a PasteScript-compatible constructor available via the egg name egg:repoze.who#config that does just this, so you can also just wire it up via a paste config file equivalently, ala:


[filter:who]

use = egg:repoze.who#config

config_file = %(here)s/etc/who.ini

log_level = debug

log_stream = stdout

You can read the documentation for more information about configuration.

- Chris

posted at: 00:00 | permalink

Authorization in Pylons with repoze.who

Tom Longson (aka nym) has created a useful blog post where he details a very simple integration between the repoze.who WSGI authentication framework and Pylons, including simple authorization.

posted at: 00:00 | permalink

2008-06-05

Six Months of Repoze

Repoze is now roughly six months old and it's a pretty good time for some self-analysis. What have we done? What are we doing? What do we want to do?

What have we done?
  • We've reimplemented the Zope 2 publisher, so Zope 2 applications such as Plone can be run without modification within a WSGI environment
  • We've successfully externalized features Zope such as transaction management, conflict error retry, virtual hosting services, and identification and authentication, error logging, and profiling into WSGI middleware, so they can be reused outside the Zope stack in a way that's largely familiar to Zope and non-Zope users alike.
  • We've packaged up Plone, Grok, Trac, and Django so they can be used without much effort in a Repoze WSGI environment.
  • We've created bridges to traditionally-CGI applications like Mailman so they can be served up via WSGI servers.
  • We've created applications like repoze.kiss, which are "Zope-like", but do not use any ZODB.
  • We've evangelized Deliverance as a theming mechanism.
  • We've reached out to other Python web projects like TurboGears, Pylons, and Django, attempting to reuse some of their components, and we've made them aware of ours.
  • We've given many, many talks at Zope/Plone/Python user groups and conferences, attempting to bring more folks into the fold
  • We've created buildouts using zc.buildout for Plone and Trac.
What are we doing?
  • We're playing around with exernalizing declarative security, so we can better integrate dissimilar WSGI applications using the same security model.
  • We're attempting to create alternate sessioning components for use under WSGI stacks.
  • We're working Repoze into our (Agendaless') own customer deployments
What we intend to do
  • We intend to do more evangelism via presenting to Python/Zope/Plone user groups and conferences.
  • We intend to reach out more to other web projects by helping them integrate Repoze code into their own projects.
  • We intend to push for Plone (and possibly Zope 2) to base a release on Repoze technologies.
Where we need help

We need help a) evangelizing, b) bringing things to ground and c) maintaining our various web presences. On the evangelism front, we need Python folks to understand that Repoze is not just about Zope, instead it's a generically useful set of WSGI components inspired by features of Zope. We need to get Zope people excited about a future that allows them to make use of best-of-breed components created by Python web developers that are not Zope developers. We need to get Python folks excited about the fact that the Zope community is reaching out to the larger Python web development community by making it possible for them to use components that Zope folks have been using for years without knowing anything whatsoever about Zope itself. We need to improve Deliverance's conditional theming support. We need Zope and Plone folks to start using repoze.zope2 and Deliverance "in anger" to start to bring things to ground. We need help maintaining repoze.org, mainly to keep content fresh. We need more developers to start contributing code to the Repoze repository.

If you want to help doing any of these things, please join the repoze-dev maillist and the #repoze IRC channel on freenode.net, and yell!

-Chris

posted at: 00:00 | permalink

Plone Symposium New Orleans Talks

posted at: 00:00 | permalink

2008-05-09

repoze.plone 2.5.5 and repoze.zope2 2.9.8 Released

For the oldskool, we've released versions of repoze.plone and repoze.zope2 which are capable of running Plone 2.5.5 and Zope 2.9.8. Previously, folks who had wished to run Plone under repoze.zope2 were limited to running either Plone 3.0.6 or 3.1.1. There are plenty of folks out there who can't move to any version of Plone 3 just yet. These releases are for you. Instructions are below about how to get Plone 2.5.5 and/or Zope 2.9.8 installed so they run under repoze.zope2. We support both buildout and virtualenv-based installs, so each is documented separately.

Installing Plone 2.5.5 (implies Zope 2.9.8)

Install repoze.plone 2.5.5 using Buildout


$ svn co http://svn.repoze.org/buildouts/repoze.plone/branches/2.5.5

$ cd 2.5.5

$ python2.4 bootstrap.py

$ bin/buildout

.. follow instructions here on in ..

... or install repoze.plone 2.5.5 using Virtualenv


$ virtualenv24 --no-site-packages plone255

$ cd plone255

$ bin/easy_install -i http://dist.repoze.org/plone/2.5.5/simple repoze.plone

$ bin/mkzope2instance .

$ bin/addzope2user admin admin

$ bin/paster serve etc/zope2.ini

Installing Zope 2.9.8 only (if you don't use Plone)

Install repoze.zope2 2.9.8 using Buildout


$ svn co http://svn.repoze.org/buildouts/repoze.zope2/branches/2.9

$ cd 2.9

$ python2.4 bootstrap.py

$ bin/buildout

.. follow instructions here on in ..

Install repoze.zope2 2.9.8 using Virtualenv


$ virtualenv24 --no-site-packages zope298

$ cd zope298

$ bin/easy_install -i http://dist.repoze.org/zope2/2.9/simple repoze.zope2

$ bin/mkzope2instance .

$ bin/addzope2user admin admin

$ bin/paster serve etc/zope2.ini

One good thing about doing this work is that it's becoming a little clearer how we need to lay out our distribution site (dist.repoze.org) to support multiple build revisions of the same piece of software. I haven't managed yet to clean it up entirely but I think I know how to now.

Have fun!

-C

posted at: 00:00 | permalink

2008-05-04

Martin Aspeli's "Rolling Out Repoze"

Martin Aspeli writes a fairly lengthy article detailing how he's moved his personal blog over to running under repoze.plone as well as configuration of another set of applications under repoze+plone + mod_wsgi + deliverance.

posted at: 00:00 | permalink

2008-04-29

Test Version of repoze.plone for Plone 3.1.1 released

To get it via easy_install (virtualenv suggested, don't even bother trying a -U upgrade from an older repoze.plone, it just won't work):


$ bin/easy_install -i http://dist.repoze.org/plone/3.1.1/simple repoze.plone

$ bin/addzope2user {username} {password}

.. ignore deprecation warnings ..

$ bin/paster serve etc/zope2.ini

To get it via zc.buildout, check out http://svn.repoze.org/buildouts/repoze.plone/branches/3.1.1/ and run:


$ python2.4 bootstrap.py

$ bin/buildout

$ bin/supervisord

$ bin/addzope2user {username} {password}

In either case, visit the result on http://localhost:8080/manage

- C

posted at: 00:00 | permalink

2008-04-27

Using repoze.who for Authentication with TurboGears 2

Since PyCon 2008 in March, we've been working within the TurboGears community on an authentication system based around repoze.who. repoze.who is a WSGI middleware authentication framework roughly modeled after Zope's Pluggable Authentication Service". repoze.who does less than PAS and shares no code with PAS, but it shares with PAS a fundamental model of a pluggable framework with customizable identification, authentication, and metadata plugins. The goal of repoze.who is to let us use the authentication features we've become accustomed to under PAS when deploying other web frameworks under WSGI. As such, repoze.who is not tied to any particular web framework. It operates as its own framework, providing downstream applications with consumable authentication and identity information. In particular, it's tied to neither TurboGears nor Zope in any way. It's tied only to WSGI, and could be used to provide similar functionality for one-off WSGI apps, Django, or Pylons, or any other WSGI-capable web framework.

The TurboGears trunk (which will at some point be released as TurboGears 2) is very different from the released 1.X version of TurboGears. Instead of using CherryPy as a controller dispatch mechanism as TG 1.X does; instead it uses Pylons. One of the side effects of this change is that the venerable TG 1.X authentication and authorization code known as identity doesn't work anymore under the TurboGears trunk. So the identification, authentication, and authorization features that TurboGears 1.X people have become accustomed to are largely missing for people working off the TurboGears trunk. An effort named Authority was established to provide the TG trunk with these features, and code exists in that project to that end. We've worked a bit with the folks who are creating Authority, and I hope there's some way to coalesce the two efforts into one in the future.

In the meantime, I created a sample set of repoze.who plugins and a repoze.who middleware configuration "on spec" for TG2 just for proof-of-concept. This eventually served as a template for Florent Aide, who subsequently developed a more-or-less feature complete set of repoze.who plugins and separate authorization facilities named tgrepozewho to provide functionality equivalent to TG 1.X's "identity" package for TG2 users. Florent also created a sample TG2 application named whotg that can make use of the authentication and authorization features provided by the tgrepozewho configuration package.

The fundamentals of Florent's TG2 application which makes use of repoze.who are these:

The result is a functional, customizable authentication and authorization system for TurboGears 2 that reuses the repoze.who framework. Hats off to Florent! This appears to be real evidence that we can move towards a "fourth generation" of Python web frameworks where coding framework-specific subsystems from scratch isn't always the norm, because the work that Florent did could be recast pretty easily for just about any TurboGears 2 or Pylons application (or Django application, etc).

- Chris

posted at: 00:00 | permalink

2008-04-16

Tomster.org: Nginx + WSGI

Tom Lazar has written an informative article on running WSGI applications under nginx. Check it out!

posted at: 00:00 | permalink

repoze.zope2 Updates

repoze.zope2 (the reimplementation of Zope2's ZPublisher that allows Zope2 to run as a WSGI application) has been updated quietly over the last few months, with the latest release, 0.3.6, posted today. Below is the changelog for the releases performed in March and April of this year. If you haven't already given it a roll, and you're interested in WSGI, you might want to try it out.




0.3.6 (2008-04-16)



  - "http" exceptions (like Redirect and Unauthorized) weren't handled

    properly anywhere except in the repoze.zope2 obob helper's

    'invoke' step.  In-the-wild code uses these exceptions before the

    published object has been located (e.g. during traverse() or

    before_traverse()).  We now depend on repoze.obob >=0.3 to get

    extended exception handling behavior, and we implement a

    'handle_exception' method on our z2bob helper which will turn

    Zope2 Unauthorized and Redirect exceptions into their

    "httpexception" equivalents for consumption by upstream

    middleware.  This was prompted by code found in the wild in

    Plone's OpenId implementation which raises a redirect during

    traversal.



0.3.5 (2008-04-16)



  - "Legacy" virtual hosting (via Virtual Host Monster) did not work

    properly.  Symptom: if you set up proxy-rewrite rules in Apache

    pointing at the Zope root a repoze.zope2 server running under a

    separate paster server, and tried to visit the ZMI via the Apache

    virtualhost's /manage URL, you'd be presented with the

    VirtualHostMonster ZMI configuration page instead of the ZMI's

    framed root UI.  Reason: the PARENTS[0] item was not set up early

    enough (it was set up in traverse rather than

    before_traverse). Since it was depended on by Zope API's which VHM

    called out to to set the virtual root, this didn't work, and the

    resulting traversal name stack was incorrect.



0.3.4 (2008-03-24)



  - Bump ez_setup.py version.



  - When Zope 2 starts, it potentially writes data to the database

    during product initialization.  When multiple clients talk to the

    same ZEO storage at startup, they often simultaneously try to

    write (the same) information to the database concurrently.  This

    causes startup failure due to conflict errors.  We now retry

    product initialization up to five times to work around this issue.



0.3.3 (2008-03-10)



  - repoze.zope2 now properly respects virtual host directives

    provided to it by repoze.vhm xheaders middleware >= 0.4.  Zope's

    VHM can still be used as necessary, but is no longer required.



0.3.2 (2008-03-03)



  - Fix bug reported by Martin Aspeli: repoze.zope2 would choke on

    large images and files (symptom: broken images when images were

    large).  This was due to the fact that the Zope File- and

    Image-rendering machinery used HTTPResponse.write, which

    repoze.zope2's response handling didn't handle properly.  We now

    subclass HTTPResponse (as RepozeHTTPResponse) to solve the issue.



0.3.1 (2008-03-02)



  - mkzope2instance now:



    o takes no arguments, only options.  '-d' replaces the single

      argument path.



    o creates a "log" directory



    o writes out a zeo.conf into "etc" (unconditionally); you can

      start a ZEO instance after installation now via 'bin/runzeo -C

      etc/zeo.conf', after ensuring that the 'address' in the ZEO

      section is correct.



    o allows the specification of 'sandbox' (-s) (replaces

      single-argument instancedir), 'zeo-port' (-z) , 'zope-port'

      (-p), and 'use-zeo' (-z) options.  If 'use-zeo' is specified,

      the zope.conf that's written will use a ClientStorage by

      default.



    o writes a zope.conf with a zodb cache-size of 50000 rather than

      10000.



  - addzope2user, runzope2script, and debugzope2 now respect a

    "ZOPE_CONF" environment variable, which can be used to specify the

    zope.conf configuration file to use.



  - Add a sample  section to the generated zope.conf

    that can be uncommented if the installer wants to use ZEO instead

    of FileStorage or vice versa.



  - Added an (experimental) 'zope2testrunner' script that sets up

    stuff in the environment before running

    'zope.testing.testrunner.run'.  It accepts the same arguments as

    the Zope 3 testrunner.  E.g. 'bin/zope2testrunner -m

    Products.faster'.  This also respects the ZOPE_CONF envvar.



  - Depend on Zope 2.10.5 (zopelib-2.10.5.0) instead of 2.10.4.2 and

    various other updated repoze libraries.

-Chris

posted at: 00:00 | permalink

2008-04-12

Tom Lazar Investigates Deliverance and Repoze

posted at: 00:00 | permalink

2008-04-09

Video of Repoze Talk by Tim Knapp

Tim Knapp of Emerge Technology presents repoze.zope2 in Christchurch NZ last week. Nice job Tim!

posted at: 00:00 | permalink

2008-04-03

Upcoming Repoze Talk Down Under

Fun for all as Tim Knapp presents Repoze in Christchurch NZ on Friday Apr. 4, 5:30 - 7:30PM. The title of the talk is The Repoze project and best practices for deployment of Python code (ie. eggs, buildout).

posted at: 00:00 | permalink

2008-03-05

Repozecast Number 1

Zopecast is dead, long live Repozecast ...

For several years now, we've been making and releasing podcasts that we've referred to as Zopecasts when we felt like it. The last few we've made have been more about Repoze than they've been about Zope, so we've decided to retitle the every-so-often podcast we do as "Repozecast".

The inaugural episode (mp3) is now up. It contains ramblings about Repoze and how it relates to:

  • The past Plone Strategic Planning Summit
  • The upcoming Plone Symposium East
  • The upcoming Python conference
  • Recent Zope/Plone/Python user group talks
  • New Repoze committers
  • New repoze.* packages
  • Buildout for repoze.plone
  • Deliverance

Enjoy!

-Chris

posted at: 00:00 | permalink

2008-02-20

repoze.profile 0.2 Released

A couple weeks ago, Tres quietly created and released a useful piece of middleware called repoze.profile. This middleware sits in the WSGI pipeline and performs aggregate Python profiling across all requests to the application(s) on its "right hand side". Ian had some profiling middleware in Paste, but it performed profiling of a single request only.

Version 0.2 adds a (butt-ugly-and-utterly-ripped-off-from-Zope) browser UI that allows you to sort and filter profiling results. It looks like the image below.

To get it, do easy_install -i http://dist.repoze.org/simple repoze.profile, then wire it up in your WSGI pipeline. For example:


 [pipeline:main]

 pipeline = egg:repoze.profile#profile

            egg:Paste#cgitb

            egg:Paste#httpexceptions

            myapp

You should then be able to visit /__profile__ in your browser while talking to your WSGI application to see the profile UI. Note that the middleware (by default) discards the first request info, so you will need to hit your application a few times to see meaningful profile info.

See the README for more detailed configuration information.

- Chris

posted at: 00:00 | permalink

repoze.profile 0.3 Released

Whoops. The earlier release of repoze.profile 0.2 today had two issues (discovered by Zac Bir, thanks Zac!).

  • It didn't work properly under Python 2.5 due to a difference in the way the Python 2.5 pstats module works
  • It didn't explicitly depend on ElementTree, which its templates require (but the templating package is not a setuptools package, so cannot name its own dependencies).

If you tried it and it didn't work, try it again.

easy_install -i http://dist.repoze.org/simple repoze.profile

Or if you've already got it installed..

easy_install -i http://dist.repoze.org/simple -U repoze.profile

- Chris

posted at: 00:00 | permalink

2008-02-18

Plone Sitetheming Part 2: Why and What

Applying a high-quality design to Plone is more of a programming task than it should be. This proposal isolates the pure look-and-feel, site-global part into something called a "sitetheme" that tailors a radically simpler, re-usable facility for corporate ID.

Motivations

Why pursue this facility? Who is it aimed at and what issues do they have with the status quo?

In a nutshell, this proposal wants to make it radically simpler to brand Plone. The designer's "sitetheming" job will focus on saving HTML to disk. No templating, with weird macro decomposition and API calls. No resource registries. No buildout. No restart.

  1. People that produce high-quality designs won't want to learn Plone templating. Imagine if their HTML (and skills therein) was used, untouched.
  2. Reducing the amount of "unique-to-Plone" stuff that is important to getting started can help reduce fear-of-the-unknown and learning curve.
  3. Sometimes this design already exists and doesn't need to be "compiled" into ZPT/packages.
  4. Web designers probably don't want to run a Plone install to maintain the the branding.
  5. Theme resources, such as CSS and JS, will be served from the HTTP server directly.
  6. Focusing Plone on the semantic information unique to a single resource/URL will promote other goals such as integration.
  7. It is possible that performance improves, if less is done in ZPT and Zope.
  8. If Plone moves to a more Python-friendly (WSGI) philosophy, then theming non-Plone sites with the same artifacts is a benefit.

Again, just to be clear: the single biggest goal is to be kind to civilians.

Sitetheming in a nutshell

First, a point about nomenclature. Since theming was introduced as a Plone word that covers skinning, templating, and (to a degree) programming, this proposal temporarily chooses a new word. We will call this new facility "sitetheming" to disambiguate jargon skew. Perhaps at the end, we'll just call this "theming".

Most pages in a site share lots of common elements. Logo, site menu links, footer, links to CSS and JS, etc. This is usually referred to "branding", "look-and-feel", or "corporate identity". On larger projects, the people managing the branding have little desire to learn a new system. On smaller projects, with people just learning, they haven't mastered enough yet to make the pages "their own".

Sitetheming is a facility for imposing a common look-and-feel on most site pages using no templating. Instead, the designer or customizer maintains a pile of HTML/CSS/JS/PNG artifacts at some reachable location. The HTML is then used as an "overlay" on the content coming out of the site.

More specifically:

  1. Save some HTML and other stuff.
  2. Make a rule file that defines spots in the theme that need to be replaced. For example, the big block in the middle column.
  3. Later, as content is being served, blocks are pulled out of the content and merged into the theme.

This diagram shows the basic idea:

In this approach, HTML is the API. As long as the theme continues providing elements with the correct ID or class names, and as long as the content coming in has a stable "REST API" of markup structure, the rule file can do its job.

Next

That covers that background and basic information. Next posts will dive into how it will actually work, plus what is needed for Plone to evaluate it as a way of doing branding.

- Paul

posted at: 00:00 | permalink

2008-02-14

Plone Sitetheming Part 1: Why Not

At the Plone Strategic Planning Summit, I signed up for work on Promote Deliverance as the branding/"corporate ID" mechanism. In the next series of blog posts, I'll try to cover the why, the what, and the how. Before beginning, though, I'd like to be up-front about some of the "why not to go in this direction."

This ticket is about putting a site's common look-and-feel on each page. For the moment, I'll call this step "sitetheming". Should we have a special, simple facility for just this audience (integrators, entry-level Plone customizers) and activity?

However, not all will agree on the basic premise, so I'd like to anticipate some of these concerns up-front. If any of this sounds like you, please speak up during this process. The goal is to make something uniquely useful and simple for the non-core-developer audience. We certainly don't want to do something that you won't like!

(If you simply must read more about site themes, here is a writeup I made long ago on the Zope wiki and here is the introduction to Deliverance. Alternatively, wait until my next few posts.)

Why Not

  1. Plone 2.5 and/or Plone 3.0 is easy enough to theme. Falls into the "ain't broke, don't fix it" category. My sense is that "theming is hard" is a common complaint by the target audience, but perhaps I'm mis-reading the tea leaves.
  2. It's hard, but worth it. Currently, theming/skinning/templating/programming are all part of the same facility. This lets you do just about anything. Creating an isolated, stripped-down "theme engine" thus breaks that proposition.
  3. I'm both the developer and the UI person. If you're mostly a developer who also does the UI, you might not want to learn a new facility, as you're still going to be the person doing the page content generation.
  4. People will invariably stray over the line. We always encourage a clear line between presentation and content. However, the moment you have something straddling the line (e.g. the contents of a calenar portlet), that discipline feels restraining.
  5. I'll never have non-Plone. As part of this proposal, we'd like to encourage moving stuff out the Zope stack, allowing it to be applied across other Python stuff. However, that might not be your itch. Having a corporate ID that could be used with other Plone apps, or potentitally non-Plone apps, might not be a meaningful upside for you.
  6. Plone needs to stop changing. Integrators, and even some core Plone folks, are feeling bumfuzzled by all the machinery changes. They've reached their absorption limit. Thus, it wouldn't matter if there was a problem, nor if this was a good solution. It's a new way to do old stuff, and that's bad on the face of it.

That's some of what I'm hearing so far. Before we get into the details of what's on the table, who should care, and even how we can minimize these points, I want to make sure I'm listening to the integrators and entry folks. Any other concerns about this proposal in particular, or the strategic planning summit in general?

- Paul

posted at: 00:00 | permalink

Continuous Integration via Buildbot

Or, How I learned to stop worrying ...

Rationale

After running across a portability / build problem with a distutils package recently, I realized that the testing we are doing for Repoze may not have adequately covered our needs to keep the various Repoze components running against all supported platforms (basically, anything where packages with "C" extensions can be built easily).

After casting around a bit, I decided to bite the bullet and try out Buildbot, the same tool used to run the Zope unit tests.

Setting Up the Master

First, I decided to set up the buildbot "master", running on the repoze.org server. This process tracks all pending changes which require builds, and doles the builds out to one or more designated "buildslaves".

Because buildbot runs as one or more long-running daemon processes, I decided to create a new account for it, to enforce a bit of privilege isolation:


  $ sudo /usr/sbin/groupadd buildbot

  $ sudo /usr/sbin/useradd -g buildbot -c "Buildbot" -m -s /bin/bash \

    buildbot

Then, I set up the new user's home directory as a virtualenv.

  $ sudo su - buildbot

  $ /usr/bin/virtualenv ~

I then installed buildbot and its setuptools-compatible dependencies (note that those dependencies aren't spelled out in its setup.py, but are document in its INSTALL.txt.


  $ bin/easy_install buildbot zope.interface

The remaining dependency is Twisted, which can't be installed via easy_install (its setup is complex, and hasn't yet adapted to the setuptools regime). So, I installed it the old-fashined distutils way:


  $ mkdir src

  $ cd src

  $ wget http://tmrc.mit.edu/mirror/twisted/Twisted/2.5/Twisted-2.5.0.tar.bz2

  $ tar xjf Twisted-2.5.0.tar.bz2

  $ cd Twisted-2.5.0

  $ ../../bin/python setup.py install

  ...

Now, I could verify that the buildbot application was working:


  $ bin/buildbot --version

  Buildbot version: 0.7.6

  Twisted version: 2.5.0

I then created and initiailzed the buildbot "master" work area, and set up a configuration (lots of iterative tweakage elided here):


  $ mkdir -p var/masters/repoze

  $ bin/buildbot create-master var/masters/repoze

  $ cp var/masters/repoze/master.cfg{.sample,}

  $ vim var/masters/repoze/master.cfg

  $ bin/buildbot start var/masters/repoze

Setting up a "buildslave"

Next, I could work on setting up the first "buildslave":


  $ mkdir -p var/slaves/repoze

  $ bin/buildbot create-slave var/slaves/repoze repoze.org:9980 \

    laguna 

I used the slavename / password which I had created in the master.cfg file. Then, I edited the two "info" files identifying me and my host, and then started the buildbot "slave" daemon:


  $ vim slave/repoze/info/{admin,host}

  $ bin/buildbot start slave/repoze

It connected, and I could see it on the status page.

Now, I tried a "forced" build, which worked (after more iterative tweakage). The "builder recipe" I'm using does the following whenever it is triggered, either manually or when its "change source" notices a change:

  1. checks out the branch containing the change (e.g., 'repoze.kiss/trunk');
  2. runs 'bin/python setup.py test' in the checkout.

The "waterfall" page shows the build status.

I then set up another pair of buildslaves, one on my Ubuntu Gutsy laptop and one on a Centos4 server, and ensured that the builds worked with them. Carlos de la Guardia has since volunteered an Ubunty Gutsy server, which is great, as my laptop may or may not be online when a change is triggered.

Conclusions

First, the product does what it is designed for: it handles the complex process of dispatching asynchronous builds to the various platforms based on changes to the repository.

Second, the configuration format is an execfiletoo powerful: I would prefer a purely declarative configuration language. Many users, however, won't need to write any really imperative code in this file, which mitigates the risk somewhat. I chose to add some code here, rather than move it out to an importable module, largely to deal with having a single "scheduler" handle changes to any of the supported branches, and to arrange that each of those branches build in a separate subdirectory.

Third, the code is definitely "frameworky", with lots of base classes and policy objects. It succeeds in hiding a lot of that from the user, however, which covers a lot of sins.

Overall, I like it for what it does, and wouldn't hesitate to use it in other settings.

Tres.

posted at: 00:00 | permalink

2008-02-12

Repoze Talk at the Seattle Plone Gathering, February 12th

Chris McDonough will be making a presentation on Repoze to the Seattle Plone Gathering this Tuesday, February 12th, starting at 6:00 PM. Thanks very much to Andrew Burkhalter for organizing the event, and especially to Ned Schumann at Olympus.net and Veda Williams, who have sponsored the travel costs.

posted at: 00:00 | permalink

2008-02-06

Package Updates, Carlos de la Guardia Updates Tutorials

New releases of repoze.plone (0.2.7), repoze.zope2 (0.3.0), repoze.grok (.0.1.5), and repoze.trac (0.3) were put up yesterday. The major change to all of them is that we made it easier to install these without relying on the "repozeproject" installer wrapper. Instead, the preferred installation mechanism is to create a virtualenv "by hand". Their respective README.txt files have the new regime documented and Carlos has kindly updated his tutorials on getting both Grok and Plone installed under mod_wsgi to reflect these new changes

The "old" installation procedure (using repozeproject) still works for the new packages, which is fortunate, as the documented install procedures on Repoze.org haven't yet been updated to describe the new installation process. I'm getting around to it, I swear... ;-)

- Chris

posted at: 00:00 | permalink

2008-02-05

Martin Aspeli Gazes into the Crystal Ball...

... and sees Repoze / Deliverance in Plone's Future

In a fascinating bit of science fiction, Martin Aspeli explores the possible user experience of Pete and Andy, the next generation of Plone newbies. His tale features both Repoze and Deliverance prominently, as the favored means for skinning and deploying Plone.

Martin then follows up by examining his flight of fancy as a roadmap forward for Plone, in particular highlighting the need for better "getting started" documentation, as well as careful attention to the usability of Plone for the "average" (non-core) developer.

Recommended reading.

posted at: 00:00 | permalink

2008-01-30

Carlos de la Guardia's repoze.grok Tutorial on grok.zope.org

Carlos has written an excellent tutorial on running a Grok application under repoze.grok and mod_wsgi. Check it out!

(Update, 2/4) Carlos blogs about this tutorial, along with another about running Plone 3 behind Apache and mod_wsgi using Repoze.

posted at: 00:00 | permalink

2007-12-23

Video of PyATL Talk Online

We just got word from Noah Gift that he has gotten the video of the first 25 minutes or so of our PyATL talk about repoze.zope2. The video is just "talking heads" (you can't see slides, or anything), but cool nevertheless.

Relevant links:

Many thanks to Noah, Derek Richardson, Brandon Craig Rhodes (who most graciously put us up in his own home), and the PyATL and Atlanta Plone folks for their hospitality. Thanks as well to all who helped sponsor travel costs, expecially to Mark Kalmes at CCP / White Wolf who matched the other contributions!

posted at: 00:00 | permalink

2007-12-18

Adding repoze.tm to a Pylons Application

After sprinting with the PyATL folks in Atlanta last week, Chris and I decided to dive into the "share Zope stuff with Python web developers" story a little more. We wanted to show that various bits of Zopeish middleware might be useful in the context of the other WSGI-enabled web frameworks.

I worked through the How to write a basic blog with Pylons tutorial, using the sqlite backend for simplicity. Once done, I looked at the code in the application which did manual / explicit transaction handline in the controller method for the blog add form POST:




    def blog_add_process(self):

        # Create a new Blog object and populate it.

        newpost = model.Blog()

        newpost.date = datetime.datetime.now()

        newpost.content = request.params['content']

        newpost.author = request.params['author']

        newpost.subject = request.params['subject']

        # I didn't set ID because it will get an autoincrement value.

        

        # Attach the object to the session.

        model.Session.save(newpost)



        # Commit the transaction.

        # (This sends the SQL INSERT command due to autoflushing.)

        model.Session.commit()

        redirect_to("/blog")



Using the transaction Framework in the Application

I decided to knock together a simple "data manager" for this application, following Chris' Transactions in WSGI tutorial. The class implements the IDataManager API, using the ORM session to do the real work:




class DataManager(object):



    transaction_manager = None



    def __init__(self, post):

        self.post = post



    def commit(self, transaction):

        """ See IDataManager.

        """

        model.Session.save(self.post)



    def abort(self, transaction):

        """ See IDataManager.

        """

        model.Session.rollback()



    def tpc_begin(self, transaction):

        """ See IDataManager.

        """



    def tpc_vote(self, transaction):

        """ See IDataManager.

        """



    def tpc_finish(self, transaction):

        """ See IDataManager.

        """

        model.Session.commit()



    def tpc_abort(self, transaction):

        """ See IDataManager.

        """

        model.Session.rollback()



    def sortKey(self):

        """ See IDataManager.

        """

        return 'myblog-sql'



I then modified the controller method such that it registers an instance of the DataManager class with the transaction. Note the addition of pseudo-validation logic, which triggers an exception in order to demonstrate the "auto-rollback" feature of repoze.tm:




import transaction

...

    def blog_add_process(self):

        # Create a new Blog object and populate it.

        newpost = model.Blog()

        newpost.date = datetime.datetime.now()

        newpost.content = request.params['content']

        newpost.author = request.params['author']

        newpost.subject = request.params['subject']



        # Register with the global two-phase transaction manager

        dm = DataManager(newpost)

        transaction.get().join(dm)



        # Trigger an error on "invalid" data, to trigger the abort.

        if newpost.subject.startswith('Abort'):

            raise ValueError('Invalid data')



        redirect_to("/blog")



Configuring the Application

First, I needed to install repoze.tm and its dependencies:




  $ ../bin/easy_install -i http://dist.repoze.org/simple repoze.tm



Then, I needed to wire the repoze.tm middleware into the PasteDeploy configuration. In order to add middleware, I renamed the [app:main] section::




[app:myblog]

use = egg:MyBlog

# Let pipeline handle errors

full_stack = false

cache_dir = %(here)s/data

beaker.session.key = myblog

beaker.session.secret = somesecret

sqlalchemy.url = sqlite:///%(here)s/db.sqlite

sqlalchemy.convert_unicode = true



Note that I turned off the full_stack option, because I want errors to propagate out to the middleware, so that it can abort the transaction.

I then defined a [pipeline:main] section, adding both the transaction middleware and some error handling::




[pipeline:main]

pipeline =

           egg:Paste#cgitb

           egg:Paste#httpexceptions

           egg:repoze.tm#tm

           myblog



At this point, the application works as desired:

  • "Normal" posts get added to the table
  • "Invalid" posts (those whose subject starts with "Abort"), are blocked.

posted at: 00:00 | permalink

2007-12-07

Highway Jam: Faster Application Restart

While driving back from North Carolina, Chris McDonough and I spent the time talking about ways to make WSGI developers more productive.

We discussed, and then discarded, the idea of trying to fix Python's reload story: it seems just too painful and error-prone (like the refresh.txt story in Zope). Instead, we realized that making restarting the application go really fast would be a big win: this observation was highlighted especially by the restart-inducde delays during the demos we did at UNc and CPCC.

In no particular order, here are the ideas we came up with:

Python-based

  • Move the current profiling support out of Zope-the-application and make it middleware, so we can measure the various bits more cleanly. Chris says Ian Bicking has some support already, but only for one request at a time.
  • When doing a "fast" restart, signal first an event which allows listeners to dump any global state to a cache on the filesystem; then, on restarting, arrange to signal another event which allows listeners to reload that state, short-circuiting the whole application initialization phase.
  • In development mode, maybe we can trigger the fast restart whenever any of [getattr(x, '__file__', None) for x in sys.modules.values()] changes. Can we somehow arrange for the triggering request to be retried after the restart? Does it matter?
    Hmm, it looks like this is what paste --reloadr does already. Cool. Score that: NIH, 0; Ian Rocks!, 1

Zope-related Changes

  • Maybe find a way to add all parsed ZCML files, to the "watched files" list too, so that changing ZCML will automagically restart the application.
  • Finish making ZCML parsing side-effect free, so that we can cache the action list (to a ".pyc"-like file) across application runs (a "pickle file").
  • Removing ZCML-parse-time side-effects implies removing the "eager validation / resolution" of dotted names. Instead, the parser would create "deferreds" representing the target objects, which would then be resolved / replaced at the point of use. We would presumably be able to serialize the deferreds in the "pickle file" mentioned above.
  • One particularly interesting bit of global state is the set of pickles cached by each ZODB connection. Repopulating those caches after a restart is one reason that Plone, for instance, takes so long to render the homepage after a restart. So, check into whether it is feasible to dump and reload the pickles maintained by each ZODB database connection to disk during a "fast restart".
  • As an alternative to dumping / reloading the ZODB Connection pickles, investigate whether multiple connections (or even appserver processes) might be able to shart them via a setup using memcached
  • Look into whether we can replace the ZEO disk cache with memcahed.
  • Measure "slow" product initializations, and find ways to speed or bypass them on restart (e.g., PlacelessTranslationService).

posted at: 00:00 | permalink

2007-12-06

Theming Trac with Deliverance

Click the screenshot images below to see them without scaling.

While in North Carolina to present Repoze to Zope and Python user groups, we worked with our host, Chris Calloway, to help wrap his trac instances within the theming and branding of his project site.

Chris supports SECOORA, a regional marine science collaboration, and needed a way to share branding for the Trac instances he maintains for the various SECOORA software projects.

Installing Trac as a WSGI app

We first worked out how to make Trac installable under Paste. Chris McDonough figured out how to get Trac installed via repoze.project, using a new dependency package, repoze.trac:


$ bin/repozeproject --path=/tmp/trac repoze.trac

Paste configuration

The standard Paste configuration that repoze.trac sets up looks like so:


$ cat etc/paste.ini

[app:trac]

paste.app_factory = repoze.trac:make_trac

path = %(here)s/../var



[pipeline:main]

pipeline = egg:Paste#cgitb

           demo

Setting up the Trac Instance

Before starting the server, we need to initialize the Trac instance in the var directory:


$ bin/trac-admin var initenv

...

Running Trac under Paste

Now we can start the server:


$ bin/paster serve etc/paste.ini

And view the Trac homepage.

Adding Deliverance to the Mix

First, install Deliverance:


$ bin/easy_install Deliverance

...

Then add a minimal rules file defining how Deliverance will merge the application into the theme:


$ cat /etc/minimal_rules.xml

<?xml version="1.0" encoding="UTF-8"?>

<rules xmlns="http://www.plone.org/deliverance" >

  <prepend theme="//head" content="//head/link" nocontent="ignore" /> 

  <prepend theme="//head" content="//head/style" nocontent="ignore" /> 

  <append theme="//head" content="//head/script" nocontent="ignore" />    

  <append theme="//head" content="//head/meta" nocontent="ignore" />

  <copy theme="//table[@id='portal-columns']"

      content="//div[@id='content']" />

</rules>

And configure Deliverance into the pipeline, using those rules:


$ cat /etc/paste.ini

...

[filter:deliverance]

paste.filter_app_factory = deliverance.wsgimiddleware:make_filter

theme_uri = http://secoora.org/

rule_uri = file:///%(here)s/minimal_rules.xml

...

[pipeline:main]

pipeline = egg:Paste#cgitb

           deliverance

           trac

Making Trac's Nav Fit into the Theme

As you can see, the minimal site does not expose any of the standard Trac navigation links: instead, its navigation is thet from the theme (a Plone site). We modified the rules file to merge the Trac links into the theme, along with the search form.


$ cat /etc/trac_rules.xml

<?xml version="1.0" encoding="UTF-8"?>

<rules xmlns="http://www.plone.org/deliverance" >

  <prepend theme="//head" content="//head/link" nocontent="ignore" /> 

  <prepend theme="//head" content="//head/style" nocontent="ignore" /> 

  <append theme="//head" content="//head/script" nocontent="ignore" />    

  <append theme="//head" content="//head/meta" nocontent="ignore" />

  <drop theme="//ul[@id='portal-siteactions']" />

  <append-or-replace theme="//head" content="//head/title" nocontent="ignore" />

  <replace theme="//form[@name='searchform']" content="//form[@id='search']" />

  <copy theme="//ul[@id='portal-globalnav']"

           content="//div[@id='ctxtnav']/ul/*" nocontent="ignore" />

  <copy theme="//ul[@id='portal-personaltools']"

           content="//div[@id='mainnav']/ul/*" />

  <append theme="//ul[@id='portal-personaltools']"

           content="//div[@id='metanav']/ul/*" />

  <drop theme="//div[@id='portal-breadcrumbs']" />

  <copy theme="//table[@id='portal-columns']"

      content="//div[@id='content']" />

</rules>

We then swtich the Deliverance Paste configuration to use the new rules.


$ cat /etc/paste.ini

...

[filter:deliverance]

paste.filter_app_factory = deliverance.wsgimiddleware:make_filter

theme_uri = http://secoora.org/

rule_uri = file:///%(here)s/trac_rules.xml

Now, we see the fully-themed site. Notice that:

  • The "tabs" in the Plone theme are now the "global" navigation links from the Trac application.
  • The "personal tools" in the theme now contains the "main" and "meta" nav links from Trac.
  • The search form is now the Trac version
  • the Plone "breadcrumbs" are gone.

posted at: 00:00 | permalink

Upcoming Repoze User Group Talks

We have beein invited to present Repoze at several Python / Zope / Plone user group meetings in December:

  • Paul Everitt is presenting a talk on Deliverance and Repoze to the D.C. ZPUG group on December 4th.
  • Chris McDonough and Tres Seaver will be doing back-to-back presentations on Repoze to the Triangle ZPUG, on December 4th, and to the Charlotte Python Group on December 5th. It looks as though we will be able to do some sprinting on the day of the 5th, as well.
  • Chris and Tres have also been invited to present Repoze to the Atlanta Python Meetup. We will be sprinting with several members of that group and te Atlanta Plone Group on Friday the 14th somewhere on the Georgia Tech camplus.

Thanks to the groups for the invitations! We are really excited to have the chance to talk about Repoze.

posted at: 00:00 | permalink

2007-12-04

Dueling Banjos: Two Repoze Presentations Tonight

Repoze fans have two chances to hear about Repoze, Deliverance wnd WSGI this evening:

Tres and Chris will recap the same talk tomorrow for the Charlotte Python Users' Group in Charlotte, North Carolina.

We hope to see as many of you as possible at one of the talks!

posted at: 00:00 | permalink

2007-11-20

0.2.8 Release of repoze.zope2

A new releases of repoze.zope2, 0.2.8, is available. This release fixes some minor redirect bugs when adding content via Zope's ZMI.

repoze.zope2 0.2.8

posted at: 00:00 | permalink

2007-11-07

Integrating Transactions into WSGI Applications Using repoze.tm

We've created a tutorial about integrating transactions into WSGI applications using repoze.tm. repoze.tm is a transaction manager implemented as WSGI middleware that uses the ZODB transaction machinery. It can be used independently of ZODB or Zope.

posted at: 00:00 | permalink

2007-11-06

"Developing With repoze.zope2"

We've just created and released Developing With repoze.zope2 which is a short 10-page PDF that explains the benefits of and differences between developing Zope 2 applications using Repoze and "stock" Zope 2. It might be helpful to folks developing under Plone and CMF. If you have comments or questions about it, please contact me or send a message to the mailing list.

Note also that I'll be introducing Repoze at the Washington DC Zope and Python User's Group tonight at 7:00pm, see the meeting description for more info.

posted at: 00:00 | permalink

2007-11-02

Mike Naberezny Posts About Repoze

Mike Naberezny, who runs Maintainable Software, posts about Repoze on his blog. Maintainable Software designed the Repoze website and is a vendor and customer for Agendaless. Mike writes production software in all of PHP, Rails, Python, TCL and JavaScript on a regular basis (I kid you not).

posted at: 00:00 | permalink

repoze.zope2 0.2.6 and repoze.plone 0.2.6 Released

We've just released repoze.zope2 0.2.6 which has the following new features:

  • WebDAV support (repoze has not supported Zope's webdav until now)
  • An error log view which can be accessed at /__error_log__ (the Zope error_log object does not function under repoze.

repoze.plone 0.2.6 is just a dependency release

If you don't already have repoze.zope2 or repoze.plone installed, you can get started at the Quickstart page. If you already have them installed, you should be able to get the latest revisions by doing easy_install -i http://dist.repoze.org/simple -U repoze.zope2 or easy_install -i http://dist.repoze.org/simple -U repoze.plone, respectively.

posted at: 00:00 | permalink

2007-10-29

In the Fitting Room: Trying on WSGI Servers

There's a funny rumor going around that somehow repoze is tied in some way to the ZServer implementation in Zope2 (or maybe its the somewhat equivalent zope.server implementation in Zope3.

Nothing could be farther from the truth: one of the driving goals for repoze is to allow interoperation with any WSGI-conformant server implementation, as well as conforming middleware. In particular, we wanted to take advantage of Graham Dumpleton's mod_wsgi, as well as the servers provided by other Python web frameworks such as the CherryPy server. In fact, this site consists of multiple WSGI applications running under mod_wsgi, as a proof of the concept.

Kicking the Tires

Back before we had actually released anything, We did some preliminary benchmarking of the various WSGI servers available.

Benchmark paste.ini File


[DEFAULT]

debug = True



[app:test]

use = egg:Paste#test



[server:paste]

use = egg:Paste#http

host = 127.0.0.1

port = 8080



[server:wsgiutils]

use = egg:PasteScript#wsgiutils

host = 127.0.0.1

port = 8080



[server:zserver]

use = egg:repoze.zope2#zserver

host = 127.0.0.1

port = 8080



[server:cherrypy]

use = egg:Paste#cherrypy

host = 127.0.0.1

port = 8080

Twisted Starter Script


# Twisted.web2 WSGI application setup

from twisted.application import service

from twisted.application import strports

from twisted.internet import reactor

from twisted.web2 import channel

from twisted.web2 import server

from twisted.web2 import wsgi

from twisted.web2.channel import http



from paste.deploy import loadapp



my_app = loadapp('config:/home/tseaver/tmp/repoze-twisted/etc/zope2.ini',

                 'zope')

resource = wsgi.WSGIResource(my_app)

site = server.Site(resource)

application = service.Application('web')

service = strports.service('tcp:8080', channel.HTTPFactory(site))

service.setServiceParent(application)

Best-Case Test

On my box, I get the following:

Twisted.Web2 ~700 req/s
wsgiutils ~900 req/s
paste ~1500 req/s
zope.server ~1500 req/s
cherrypy ~2200 req/s

I ran the twisted server with the attached server descriptor:


  $ bin/twistd -ny bin/twisty.py

The others I ran under paster:


  $ bin/paster --server-name= --app-name test etc/zope2.ini

The endpoint in each case is egg:Paste#test (I wanted 'pony', but Chris talked me out of it).

Zope Quickstart

Running against the Zope quickstart page (editing the name in 'twisty.py', switching --app-name to 'zope'):

Twisted.Web2 ~160 req/s
wsgiutils ~165 req/s
paste ~220 req/s
zope.server ~225 req/s
cherrypy ~240 req/s

Tests run against all settings via:


  $ ab -n 1000 -c 10 http://localhost:8080/

References

The original mailing list thread where we tried this stuff out:

posted at: 00:00 | permalink

2007-10-28

Repoze Presentation at ZPUG-DC November 6, 2007

I (Chris McDonough) will be presenting an introduction to Repoze at a meeting of the Washington DC Zope and Python User's Group on Tuesday, November 6 2007 in the Radio Free Asia boardroom . Sadly this is also election day. If you should need to choose between coming to see this presentation or voting, you should vote instead. ;-)

For more information, see the ZPUG DC announcement.

posted at: 00:00 | permalink

2007-10-27

repoze.zope2 0.2.5 and repoze.plone 0.2.5 Released

Plone's Kupu and KSS Javascript wasn't working up to 0.2.5, now it is. You can update to the newest release by doing (in your sandbox):


$ bin/easy_install -U repoze.zope2

posted at: 00:00 | permalink

Martin Aspeli Cheers On Repoze

Martin Aspeli, better known in Ploneland as "optilude", has a new blog post up which is really favorable to Repoze. Check it out!

posted at: 00:00 | permalink

2007-10-25

Screencast Two: Setting Up Deliverance

Want to theme Plone without learning the stack? Want to use an existing HTML look-and-feel, without touching it, to give the basic style to your content...Plone or otherwise?

Screencast 2: Setting Up Deliverance shows you how to use WSGI middleware and boring old web standards to theme your content. (3.6 Mb, 3m18s duration, Quicktime 7-ish.) This demo shows:

  • Inspecting the theme and the content for merge points
  • Editing the Paste config file to insert the WSGI middleware (Deliverance). One word plus 3 lines of change!
  • Writing a simple rules file to merge the Plone content into the theme.
  • Restarting the server and seeing your newly-themed site.

I have waited a long, long time to be able to give this screencast. Years ago, in the original Zope 3 wiki, I wrote a proposal for Zope Site Themes. I worked on a couple of mod_python implementations. Then Ian and Luke at OpenPlans picked up the idea and converted it from a useless toy into a real-deal software package as WSGI middleware.

Unfortunately there was always a piece missing. If you wanted to do Plone, you wanted to do Zope2. And that mostly meant no WSGI. Ian had a publisher hack you could use, but it wasn't really a mainstream option. And not everyone would be willing to roll up their sleeves and get mod_python going.

With Repoze, Chris and Tres have now made all of this not just possible, but probable.

As an example, we spent money on a theme produced by some cools guys to the west of here. They delivered HTML, CSS, etc., just as the Deliverance theory anticipates. Instead of hacking into a ZPT with macros/slots, we used it as-is, fronting:

  • www.repoze.org, a WSGI app that converts STX on the filesystem on-the-fly using a Zope-publisher-like thing
  • blog.repoze.org, running pyblosxom
  • bugs.repoze.org, running Roundup
  • And soon, demo.repoze.org (a Plone) and lists.repoze.org (Mailman) will get themed

All of these use exactly the same, untouched HTML/CSS/JS/PNG for the theme. It took Tres around 20 minutes to "theme" Roundup...without touching Roundup either!

Me likey.

posted at: 00:00 | permalink

Screencast 1: Installing Repoze

Want to watch how to get started with Repoze? 7.5 Mb, 3m33s of viewing, and a fairly recent QuickTime installation is all you need for Screencast 1: Installing Repoze.

This screencast shows:

  • Snipping the right commands from the Repoze site
  • Using easy_install to get a repozeproject script
  • Running repozeproject to produce a Paste-driven, WSGI/virtualenv Zope2/Plone install. (To paraphrase Team America, Hell Yeh!.)
  • Using the easy_install in your virtual Python to add Deliverance.

Notes and caveats:

  • Sure, I should write a bit more material on this. I plan to, honest. I know some folks (see Geir, I'm listening to you!) prefer reading and static images to movies. I'd like to produce a guide that combines both. But no need to hold this up until perfection arrives.
  • Not a lot of titling. I decided to try dumping Camtasia Studio, which meant authoring on Windows. Instead, I used a Mac toolchain: iShowU -> iMovie 08. Let's just say the Steve giveth and the Steve taketh away. Perhaps I'll brush things up later.
  • My plan is to have a series of screencasts (and text) that show how to really-really move theming out of Plone and into Deliverance. Not just replace what Plone does, but instead, make a Plone skin that doesn't do it in the first place.

posted at: 00:00 | permalink

2007-10-24

Announcement: repoze.tempita 0.2 Released

repoze.tempita is a very early cut at creating a middleware filter which causes Tempita syntax returned by a downstream application to be conditionally expanded using replacement configuration within a paste.deploy file

For more information, see the repoze.tempita README.txt file

To install, do bin/easy_install -f http://dist.repoze.org repoze.tempita

posted at: 00:00 | permalink

2007-10-22

Announcement: repoze.zope2 0.2.3 and repoze.plone 0.2.3 Released

repoze.zope2 0.2.3 and repoze.plone 0.2.3 now use repoze.project

You can (indeed, must) now install repoze.plone or repoze.zope2 sandboxes using repoze.project. As of this release the old "setup.py sandbox" sandbox creation mechanism is no longer supported in these packages.

Here's an example of using repoze.project to install repoze.plone




 $ bin/easy_install -i http://dist.repoze.org/simple/ repoze.project

 ...

 $ bin/repozeproject --target-path=/tmp/ptest repoze.plone



This will cause a sandbox to be create in /tmp/ptest including all "instance" files (such as etc/zope.conf, etc).

posted at: 00:00 | permalink

2007-10-20

Announcement: repoze.project 0.0.2 Released

What is repoze.project?

repoze.project is a Python distribution which installs a script, repozeprojectct, into your Python environment. Ths intent of the script is to make it simple to create a new repoze application deployment environment, in somewhat the same manner as zopeproject does for Zope3, and as grokproject does for Grok applications.

However, rather than using Jim Fulton's zc.buildout to drive the creation of a new environment, we chose to use Ian Bicking's virtualenv, because we like the concept of "standalone" environments with custom software in their own site-packages directories.

Trying it Out

As the version number implies, this is a pretty early version of the repoze.project. Neverthless, the package does enable the following method of installing Plone-under-repoze:




 $ bin/easy_install -i http://dist.repoze.org/simple/ \

   repoze.project

 ...

 $ bin/repozeproject --target-path=/tmp/ptest repoze.plone



One caveat about the example: the repoze.plone project has not yet been updated to add the entry point which repoze.project looks for when initizlizing the environment.

posted at: 00:00 | permalink

2007-10-18

Announcement: repoze.grok 0.1 Released

We've been working on getting a repoze wrapper for Zope3 apps, and Grok in particular, up and running.

I'm pleased to say that we have a distribution available for testing.

Installing from the Tarball


 $ wget http://dist.repoze.org/repoze.grok-0.1.1.tar.gz

 $ tar xzf repoze.grok-0.1.1.tar.gz

 $ cd repoze.grok-0.1.1

 $ ../../bin/python setup.py sandbox \

   --admin-login=admin --admin-password=123

 ...

 $ bin/paster serve etc/grok.ini

Installing via easy_install


 $ easy_install -f http://dist.repoze.org repoze.grok

 $ bin/init_configs --admin-login=admin --admin-password=123

 $ mkdir var  # directory for the ZODB.

 $ bin/paster serve etc/grok.ini

Browsing the Application

You can then connect to the Grod Admin UI. Of course, there won't be any applications you can add (yet), but you can browse the application control UI and the documentation.

Adding a Grokkified Application

You can download and install one of the sample grok apps, or even roll your own based on the tutorial examples.

posted at: 00:00 | permalink

2007-10-13

New Releases of repoze.zope2 and repoze.plone

New releases of repoze.plone and repoze.zope2 are available. The latest releases (0.2.2, for both), allow usage of the Plone login portlet (which did not work up until this release).

posted at: 00:00 | permalink

2007-10-03

IRC Channel for Repoze

We've set up an IRC channel for discussing repoze developement, and answering questions:

irc://irc.freenote.net/#repoze

posted at: 00:00 | permalink

2007-10-01

Announcing Repoze

"Whisky. Champagne. Mickey's."

But seriously.... This blog is for news and updates concerning the Repoze project.

Repoze is a refactoring / rewrite of portions of the Zope application server, to make it possible to run Zope and Zope-like applications behind a WSGI-compliant server.

Other Repoze Resources

posted at: 00:00 | permalink

RSS Feed