Mapreduce实例-分组排重(group by distinct)

发布时间:2016-12-10 11:34:36 编辑:www.fx114.net 分享查询网我要评论
本篇文章主要介绍了"Mapreduce实例-分组排重(group by distinct)",主要涉及到Mapreduce实例-分组排重(group by distinct)方面的内容,对于Mapreduce实例-分组排重(group by distinct)感兴趣的同学可以参考一下。

需要实现以下几个类,代码太多,列了下主要代码,可根据排重数据的特征判读是否需要添加combiner来提速。 public class GroupComparator implements RawComparator<MyBinaryKey> { @Override public int compare(MyBinaryKey o1, MyBinaryKey o2) { return o1.toString().compareTo(o2.toString()); } @Override public int compare(byte[] b1, int s1, int l1, byte[] b2, int s2, int l2) { return WritableComparator.compareBytes(b1, s1, Long.SIZE / 8 + Integer.SIZE / 8 * 3, b2, s2, Long.SIZE / 8 + Integer.SIZE / 8 * 3); } } public abstract class UVBinaryKey extends BinaryComparable implements WritableComparable<BinaryComparable>{ //根据需要添加属性; @Override public void readFields(DataInput in) throws IOException { } @Override public byte[] getBytes() { } } public class MyPartitioner extends Partitioner<MyBinaryKey, NullWritable> { /** * 根据uv/ip取模分区,保证相同uv/ip落在同一分区 */ @Override public int getPartition(MyBinaryKey key, NullWritable value, int numPartitions) { int k=0; for(byte b : key.getAttr()){ k+=b&0xff; } return k%numPartitions; } } job.setMapOutputKeyClass(UVBinaryKey.class); job.setGroupingComparatorClass(GroupComparator.class); job.setPartitionerClass(MyPartitioner.class); map略combiner(根据需要添加) reduce中的实现: @Override protected void reduce(UVBinaryKey key, Iterable<NullWritable> values, Context context) throws IOException, InterruptedException { long count = 0; byte[] tbsign = null; for (NullWritable nullWritable : values) { byte[] attr = key.getAttr(); if (tbsign == null) { tbsign = attr; count++; } if (tbsign != null) { if (tbsign.length != attr.length) { count++; tbsign = attr; } else { for (int i = 0; i < tbsign.length; i++) { if (tbsign[i] != attr[i]) { count++; tbsign = attr; break; } } } } } StringBuffer out = new StringBuffer(); out.append(new String(key.getCity())) .append(Constants.FIELDS_TERMINATED).append(count); context.write(new Text(out.toString()), NullWritable.get()); }     

上一篇:windows下的环境搭建配置redis
下一篇:JSP和Servlet中的几个编码的作用及原理

相关文章

相关评论