2010年8月31日星期二

通过ibatis实现轻量级的水平切分

通过ibatis实现轻量级的水平切分: "最近想在自己的项目里实现DB sharding功能,正好前段时间研究过ibatis的源码于是就在ibatis的基础上进行了一些修改。另一方面也是为了练练手。这个sharding的实现主要是基于我项目中的需求实现的可能有很多考虑不周的地方,希望各位大牛拍砖。如果有人感兴趣愿意一起来发展这个项目,本人也非常欢迎各位的加入。

Shardbatis是在mybatis 2.3.5代码的基础上进行一些扩展实现数据水平切分功能。 数据的水平切分包括多数据库的切分和多表的数据切分。目前shardbatis已经实现了单数据库的数据多表水平切分

mybatis2.3.5的核心类图(包含了spring对ibatis的封装SqlMapClientTemplate)如下(其他版本的ibatis的类图有略微不同)



改造后的类图



从这两张图上可以看出shardbatis里新增了接口SqlMapShardingExt,SqlMapShardingExt中具体的方法如下


public interface SqlMapShardingExt extends SqlMapExecutor{
/**
* 带有sharding功能的insert
* @param id
* @param parameterObject
* @param groups
* @return
* @throws SQLException
*/
Object insertWithSharding(String id, Object parameterObject,
ShardingFactorGroup... groups) throws SQLException;

/**
* 带有sharding功能的insert
* @param id
* @param groups
* @return
* @throws SQLException
*/
Object insertWithSharding(String id, ShardingFactorGroup... groups) throws SQLException;

/**
* 带有sharding功能的update
* @param id
* @param parameterObject
* @param groups
* @return
* @throws SQLException
*/
int updateWithSharding(String id, Object parameterObject, ShardingFactorGroup... groups)
throws SQLException;

/**
* 带有sharding功能的update
* @param id
* @param groups
* @return
* @throws SQLException
*/
int updateWithSharding(String id, ShardingFactorGroup... groups) throws SQLException;

/**
* 带有sharding功能的delete
* @param id
* @param parameterObject
* @param groups
* @return
* @throws SQLException
*/
int deleteWithSharding(String id, Object parameterObject, ShardingFactorGroup... groups)
throws SQLException;

/**
* 带有sharding功能的delete
* @param id
* @param groups
* @return
* @throws SQLException
*/
int deleteWithSharding(String id, ShardingFactorGroup... groups) throws SQLException;

/**
* 带有sharding功能的单记录查询
* @param id
* @param parameterObject
* @param groups
* @return
* @throws SQLException
*/
Object queryForObjectWithSharding(String id, Object parameterObject,
ShardingFactorGroup... groups) throws SQLException;

/**
* 带有sharding功能的单记录查询
* @param id
* @param groups
* @return
* @throws SQLException
*/
Object queryForObjectWithSharding(String id, ShardingFactorGroup... groups)
throws SQLException;

/**
* 带有sharding功能的单记录查询
* @param id
* @param parameterObject
* @param resultObject
* @param groups
* @return
* @throws SQLException
*/
Object queryForObjectWithSharding(String id, Object parameterObject,
Object resultObject, ShardingFactorGroup... groups)
throws SQLException;

/**
* 带有sharding功能的查询
* @param id
* @param parameterObject
* @param groups
* @return
* @throws SQLException
*/
List queryForListWithSharding(String id, Object parameterObject,
ShardingFactorGroup... groups) throws SQLException;

/**
* 带有sharding功能的查询
* @param id
* @param groups
* @return
* @throws SQLException
*/
List queryForListWithSharding(String id, ShardingFactorGroup... groups)
throws SQLException;

/**
* 带有sharding功能的查询
* @param id
* @param parameterObject
* @param skip
* @param max
* @param groups
* @return
* @throws SQLException
*/
List queryForListWithSharding(String id, Object parameterObject, int skip, int max,
ShardingFactorGroup... groups) throws SQLException;

/**
* 带有sharding功能的查询
* @param id
* @param skip
* @param max
* @param groups
* @return
* @throws SQLException
*/
List queryForListWithSharding(String id, int skip, int max,
ShardingFactorGroup... groups) throws SQLException;

/**
*
* @param id
* @param parameterObject
* @param rowHandler
* @param groups
* @throws SQLException
*/
void queryWithRowHandlerWithSharding(String id, Object parameterObject,
RowHandler rowHandler, ShardingFactorGroup... groups)
throws SQLException;

/**
*
* @param id
* @param rowHandler
* @param groups
* @throws SQLException
*/
void queryWithRowHandlerWithSharding(String id, RowHandler rowHandler,
ShardingFactorGroup... groups) throws SQLException;

/**
*
* @param id
* @param parameterObject
* @param keyProp
* @param groups
* @return
* @throws SQLException
*/
Map queryForMapWithSharding(String id, Object parameterObject, String keyProp,
ShardingFactorGroup... groups) throws SQLException;

/**
*
* @param id
* @param parameterObject
* @param keyProp
* @param valueProp
* @param groups
* @return
* @throws SQLException
*/
Map queryForMapWithSharding(String id, Object parameterObject, String keyProp,
String valueProp, ShardingFactorGroup... groups)
throws SQLException;


通过SqlMapClientImpl和SqlMapSessionImpl对SqlMapShardingExt的实现是的ibatis具有了DB sharding的功能,iBATIS会自动根据配置的或者是编码的sharding策略将原始的sql语句转变为对应目标表名的sql。

下面看一下如何在iBATIS中配置和使用sharding功能

1.配置sharding策略。在sql-map-config.xml中添加如下配置


<shardingConfig>
<!-- 对于app_test使用默认的切分策略,默认的切分策略只是简单按数值取模生成新的表名,如何实现切分策略后面再介绍 -->
<sharding tableName="app_test" strategyClass="com.ibatis.sqlmap.engine.sharding.impl.DefaultShardingStrategy"/>
<!-- 没有配置切分策略等于使用默认的切分策略 -->
<sharding tableName="app_2test" />
</shardingConfig>


这里再通过实例简单介绍一下使用DefaultShardingStrategy时对sql的convert结果

比如sqlmap中定义的原始sql为:


SELECT EMPLOYEEIDNO FROM mytable WHERE SALARY >= 50000


那经过convert后的结果将可能是


SELECT EMPLOYEEIDNO FROM mytable_1 WHERE SALARY >= 50000


又例如原始sql为


SELECT a.* FROM ANTIQUES a,ANTIQUEOWNERS b, mytable c where a.id=b.id and b.id=c.id


convert的结果可能为


SELECT a.* FROM ANTIQUES_0 AS a, ANTIQUEOWNERS_1 AS b, mytable_1 AS c WHERE a.id = b.id AND b.id = c.id


开始使用sharding api


public class SqlMapClientTest {
SqlMapClient sqlMapper;

@Before
public void init() {
Reader reader;
try {
reader = Resources.getResourceAsReader('sql-map-config.xml');
sqlMapper = SqlMapClientBuilder.buildSqlMapClient(reader);
reader.close();
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

@Test
public void testSharding() throws Exception {
Map param = new HashMap();
param.put('cnt', 'ttt');
ShardingFactorGroup g = new ShardingFactorGroup();//ShardingFactorGroup为切分策略提供必要参数
g.setTableName('App_Test');//设置为哪张表配置切分策略
g.setParam(new Integer(123));//设置为切分策略提供参数
//这里还可以通过g.setShardingStrategy(...)来设置切分策略
//通过API配置的切分策略可以覆盖xml里配置的App_Test表的切分策略
Integer count = (Integer) sqlMapper.queryForObjectWithSharding(
'AppTest.select_paging_count_by_map', param, g);
Assert.assertEquals(count.toString(), '0');
}

@Test
public void testUpdate() throws SQLException {
ShardingFactorGroup g = new ShardingFactorGroup();
g.setTableName('App_Test');
g.setParam(new Integer(123));

String cnt = 'testUpdate' + System.currentTimeMillis();
AppTest at1 = new AppTest();
at1.setCnt(cnt);
Integer id = (Integer) sqlMapper.insertWithSharding(
'AppTest.insert_h2', at1, g);

AppTest parameterObject = new AppTest();
parameterObject.setCnt(cnt);
AppTest ret = (AppTest) sqlMapper.queryForObjectWithSharding(
'AppTest.select_by_condition', parameterObject, g);
Assert.assertEquals(ret.getId().toString(), id.toString());

ret.setCnt('NEW_CONTENT');
Integer count = sqlMapper.updateWithSharding('AppTest.update', ret, g);
Assert.assertEquals(count.toString(), '1');

count = (Integer) sqlMapper.queryForObjectWithSharding(
'AppTest.select_paging_count', ret, g);
Assert.assertEquals(count.toString(), '1');
}

@Test
public void testDelete() throws SQLException {
ShardingFactorGroup g = new ShardingFactorGroup();
g.setTableName('App_Test');
g.setParam(new Integer(123));

String cnt = 'testDelete' + System.currentTimeMillis();
AppTest at1 = new AppTest();
at1.setCnt(cnt);
Integer id = (Integer) sqlMapper.insertWithSharding(
'AppTest.insert_h2', at1, g);

AppTest parameterObject = new AppTest();
parameterObject.setCnt(cnt);
AppTest ret = (AppTest) sqlMapper.queryForObjectWithSharding(
'AppTest.select_by_condition', parameterObject, g);
Assert.assertEquals(ret.getId().toString(), id.toString());

Integer row = sqlMapper.deleteWithSharding('AppTest.delete', ret, g);
Assert.assertEquals(row.toString(), '1');
}
}


实现自己的sharding策略,只要实现一个简单的接口即可


public interface ShardingStrategy {
/**
* 计算得到新的表名
* @param baseTableName 逻辑表名
* @param params 为sharding逻辑提供必要参数
* @return
*/
public String getTargetTableName(String baseTableName,Object params);
}


关于shardbatis在spring中的使用方法,以及一些使用注意事项和性能测试结果请大家移步到项目主页http://code.google.com/p/shardbatis/上查看




作者: SeanHe 


声明: 本文系JavaEye网站发布的原创文章,未经作者书面许可,严禁任何网站转载本文,否则必将追究法律责任!




已有 1 人发表回复,猛击->>这里<<-参与讨论





JavaEye推荐






"

没有评论:

发表评论